Evaluation ICCO Alliance Programmatical Approach

Evaluation of the ICCO Alliance Programmatic Approach

ANNEXES INCLUDING THE TERMS OF REFERENCE AND THE THREE SUB-REPORTS

TABLE OF CONTENT

Annex 1 -­‐ Terms of Reference Advancement of ICCO Alliance’s working with the Programmatic Approach 2009 – 2012

Annex 2 – Work plan ICCO Evaluation Programmatic Approach

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach -­‐ Survey about the development and performance of coalitions

Annex 5 – Terms of Reference Advancement of ICCO Alliance’s working with the Programmatic Approach 2009 – 2012, phase 3, field study Annex 6 -­‐ Report of the field (third) phase of the Evaluation Programmatic Approach

Annex 1 -­‐ Terms of Reference Advancement of ICCO Alliance’s working with the Programmatic Approach 2009 -­‐ 2012 1. Introduction This terms of reference concerns the second evaluation of The ICCO Alliance Programmatic Approach. A first evaluation has been carried out by two external consultants, Erica Wortel and Jouwert van Geene, and was mainly based on a desk study. The evaluation had a strong focus on learning directed to provide input to further strengthen the implementation of the Programmatic Approach. Many of the recommendations have been taken up in the new Business plan 2011 – 2015.

The Programmatic Approach still uses very much a ‘learning by doing’ approach, with changes being induced by experiences in the ‘field’ and reflections on the way of working with the approach. After another 3 years of working with the PA, the ICCO Alliance wants to take a more in-­‐depth look again and see what the issues are at this moment, how the PA is working, and if tangible results can be seen already?

2. Background The ICCO Alliance is implementing a Programmatic Approach (PA) since the start of its first Business Plan 2007 -­‐2010 in 2007. PA is both part of ProCoDe, the change agenda of ICCO focussing on decentralisation and co-­‐responsibility, and an important supporting1 instrument in this change agenda. The organization has chosen to implement the PA in a ‘learning by doing’ and ‘continuous adaptation’ approach, so step by step and with changes induced by own learning on the use of the PA in practice.

Theory of Change of the Programmatic Approach The Programmatic Approach is essentially about the way in which the ICCO Alliance2 promotes cooperation between organizations in developing countries in order to reach development results.

As ICCO Alliance we recognize that poverty and injustice are always related to complex problems in which many people have a stake and where organizations represent certain interests. All are embedded in larger systems that often maintain existing inequalities. The combination of different systems makes up societies. The ICCO Alliance aims at changing the systems that embed and maintain inequalities in such a manner that poverty is ended, justice is guaranteed and rights of all individuals and communities are respected. To be able to do so we propose to work in an approach that will support actors with different stakes in systems to come together and develop a shared agenda for change. We call this approach the Programmatic Approach which is defined as follows:

A multi stakeholder process that leads to organizations working together based on a joint analysis, shared vision and objectives and clear perspective on the results of the cooperation. In such a process all actors can do different things, work at various levels and use their own strengths for the common purpose and objectives, as well as share some activities and in particular share and participate in the linking and learning processes. The programmatic approach does not only address single problems but aims at change in systems3

The Programmatic Approach is not only a new strategy for the ICCO Alliance. It is also a very specific form of Programmatic Approaches. Programmatic Approaches are mainly interpreted as the bringing 1

The International Advisory Group phrased this support of PA in 2007 as stimulus for: complementary and cooperative roles between stakeholders; a common goal; a participatory process and co-owned decision making 2 The ICCO Alliance is formed by: ICCO, Edukans, Prisma, KerkinActie, Share People, ZeisterZendingsgenootschap, Oikocredit, Yente 3 A system is a set of interacting or interdependent entities forming a larger whole. These systems may include organisational systems, may have geographical boundary, often has multiple levels and actors. Systems have the capacity to change, to adapt when it is necessary in response to internal or external stimulus. Complex Adaptive Systems, Heather Baser and Peter Morgan, Complex Adaptive Systems Theory, ECDPM 2004

Subject:ToR Programmatic approach status:draft Author: Dieneke de Groot; Hettie Walters

2009 - 2012

3 date: April 2013

together of relevant projects and programs in one planning container such as the Sectoral Approach.

For the ICCO Alliance the core of the Programmatic Approach is that we support cooperative processes of multiple stakeholders aiming at creating systemic change because we think that we need the strength of cooperation to be effective in realizing difficult and complex changes. It is therefore not mainly a planning approach for the ICCO Alliance but a strategy for realizing fundamental change with our partner organizations and other stakeholders in the areas in which we work.

This angle is offered by Systems Thinking. Systems are defined as interactions among diverse agents that persist and evolve as a coherent whole. Systems Thinking looks at the ‘whole’ first and examines how parts of the wider whole influence each other, or change as result of their relationship to their environment. Attention to the various elements of the system is secondary to attention to the whole4 5. Systems thinking states that changes in parts of a system will always cause the whole system to change. This change will however not have a predictable result nor can it be planned in a linear fashion. The ICCO Alliance takes systems behavior into account in its Programmatic Approach.

The insecurity that is implied by the unpredictability of changes needs to be reflected in the monitoring and evaluation systems that we use. In addition to measuring expected changes, we need to be able to capture the unexpected and ‘notice’ emergent change as well.

This line of thinking has resulted in the following theory of change underpinning our Programmatic Approach: • Development problems are the result of complex systems of interlinked actors, structures, institutions and processes • Complex problematics demand an approach that can deal with and work in the complexity.

Therefore a Multi Stakeholder Process (MSP) is needed • MSPs lead to joint learning and cooperation between the actors involved • The MSP represents the system involved in the problematic. Cooperation between actors and organizations leads to added value: greater effectiveness in change at the institutional level and whole system change.

• The ICCO Alliance will support existing cooperative processes and initiate the cooperative process if none exists yet.

• Coalitions of cooperating actors have (and adhere to) ownership in the programmatic cooperation (the program).

• This also implies that a coalition can identify possibilities for diversification of funding sources to assure sustainability of the cooperation and independence from the ICCO Alliance. It is preferable that the cooperative process is not solely dependent on ICCO Alliance funding.

Evaluation In 2009 an evaluation of the programmatic approach of the ICCO Alliance was carried out. This evaluation – although very relevant-­‐ was done in a time that the PA had not yet landed fully in the organization (and the decentralized organization later on) and the decentralization itself was not yet fully implemented. Both factors were certainly contributing to confusion on the content of this new way of working. The evaluation yielded useful recommendations to improve working along the PA principles which were taken up by the organization. Main recommendations were to: - ‘recalibrate’ the Programmatic Approach and give more theoretical underpinning - close the learning loops on the Programmatic Approach in the organization - strengthen capacity for the Programmatic Approach on all levels: ROs – GO and program coalitions 4

Definition by Peggy Holman in Engaging with Emergence, page 220, Berrett Koehler 2010 The idea and practice of systems thinking and their relevance for capacity development’, Peter Morgan, ECPDM march 2005 5

Subject:ToR Programmatic approach status:draft Author: Dieneke de Groot; Hettie Walters

2009 - 2012

4 date: April 2013

- provide instrument to support the approach in MSD and instruments

PA capacity development Already starting under the first Business Plan various types of support have been given to various bodies in the organization to draw PA to a higher level. In this support the lessons drawn from working in a programmatic way in practice are being taken along. The table below gives an overview of the various learning and capacity development activities around the Programmatic Approach:

Table Overview of PA activities Year Activity For whom by whom Follow-­‐up 2008-­‐ Training of staff of Staff ICCO CDI Open Space 2009-­‐ ICCO in Utrecht and Staff ICCO Alliance sessions 2010 staff of ICCO Alliance member partners organisations NL 2008-­‐ Open Space learning Staff ICCO Alliance P&D Appreciating and 2009-­‐ sessions NL on-­‐going training 2010 programmes 2008-­‐ Learning History Programmes and Hester Pronk Appreciating and 2009 development staff ICCO Alliance on-­‐going training programmes 2009-­‐ Appreciating the 6 programmatic Appreciating Synthesis document 2010 Programmatic Development team: James recommendations, Approach processes Taylor developing Pscan Kaustuv Bandhopadyay Meas Nee Domien Bruinsma Lisette Caubergs Charly Buxton Fons van der Velden Hettie Walters 2011-­‐ Regional training on POs, partner Hettie Walters Recommendations 2012 PA in Central Asia, organizations to staff and partners Central America, South Africa and Southeast Asia 2012 5 Action Researches on Programmes and Junior action Reports for RO; film state of affairs of 5 staff ICCO Alliance researchers of feed back programmes session, films on the use of participatory video processes in 2 programmes 2012 Participation in PSO Two programs: Penpen Libres ( Reports, filmed TLP on Power in Multi-­‐ organisations and Philippines interviews, joint Stakeholder Processes stakeholders Seeweeds VCD) publication and Job Blijdenstein (Forestry program, Guatemala) 2011-­‐ 3 Continental Training ICCO staff and Hettie Walters and Strengthening ToC Subject:ToR Programmatic approach status:draft Author: Dieneke de Groot; Hettie Walters

2009 - 2012

5 date: April 2013

2012

of Trainers on the Programmatic Approach

Training ROs on PA Kirgystan, South –East Asia Training on ToC

2012-­‐ 2013

Partner organisations 40-­‐ 60% >100 participants PO

CDI

Approach in PA

Hettie Walters

Ongoing, linked to many other CoL and training/ workshops where we meet staff ICCO and partners and in WASH Alliance

Hettie Walters

In some Programs ToC are being developed.

The lessons6 learned of both the Action Research and the Thematic Learning Programme have been merged by the Capacity Development Coordinator in several synthesis documents and in the Guidance on the Programmatic Approach note. Processes have been documented and final products have all been placed on the Compart wiki: Programmatic Approach. The Action Research has led to a final document that was shared, such as a video of the final presentation and a blog on the development of ToC in one of the programmes involved. These products and other recent products of learning processes will be posted on the new ICCO portal Programmatic Approach community.

After all these inputs on strengthening the PA and another 3 years working in practice with this approach the ICCO Alliance wants to undertake a new evaluative study. This is fully in line with the intention to become a learning organization. The outcomes of the evaluation will serve as input for further learning on working with the PA as methodology and possibly leads to changes and adaptations in the ways of working for the coming years till 2015 and possibly beyond.

3. Purpose 3.1. Purpose Purpose of this evaluation is to further deepen our understanding of the Programmatic Approach in practice and its contribution to the implementation of effective programs reaching their (change) objectives. The main focus is on critical aspects of the Programmatic Approach like ownership, stakeholder diversity and sustainability, and to grasp the factors related to these issues. This evaluation purposefully builds on the various (internal) studies done on the Programmatic Approach by giving a comprehensive external view.

With the results of the evaluation the ICCO Alliance will: -­‐ have deeper understanding in the development of the Programmatic Approach, both in relation to the development and functioning of the programmatic coalitions and the way in which program coalitions work towards reaching their development objectives.

-­‐ get pointers for further consolidation of the Programmatic Approach

6

Main issues from the Action Research and the Thematic Learning Programme are: How are partners cooperating, and how has their joint ‘programme’ been developed, how is monitoring and joint learning taking place? What is the role played by the ICCO Alliance in the Programmatic cooperation? In the action research the final recommendations all concerned the creation of clarity: about purpose of the cooperation, of envisaged results of the cooperation, of roles played by all actors involved, clarity in developing a Theory of change, clarity on responsibilities of all actors involved, and in particular about ownership in relation to funding.

Subject:ToR Programmatic approach status:draft Author: Dieneke de Groot; Hettie Walters

2009 - 2012

6 date: April 2013

-­‐ -­‐ -­‐

gain insight in the contribution of the Programmatic Approach to the realization / implementation of ESP3. gain insight in the role the ICCO Alliance plays in strengthening-­‐ hindering the development of fruitful cooperative processes. gain insight in particular in the relation between the Programmatic Approach and the other crosscutting approaches : Gender and the Rights-­‐based approach: are programmes and coalitions working in an inclusive manner as to gender and human rights?

3.2 Objective The objective of the evaluation of the Programmatic Approach is to get insights on some more qualitative and some quantitative questions regarding the programmatic approach. The outcomes of the evaluation will help to further consolidate the Approach for the remaining period of the business plan 2011 -­‐ 2015 and for policy development beyond 2015.

The most urgent questions on this moment are: “What works well and what lags behind or doesn’t work, for what reason in relation to the Programmatic Approach?”; Other urgent questions are questions around emerging partnerships and their development: “How has the programmatic coalition taken shape?”; “Who are the stakeholders in the program coalition?” and ”Who has the ownership of the programmatic coalition?”

Four topics are identified around issues which are considered to be crucial for further consolidation of the Programmatic Approach. These are; Actors/ organisations involved in programme coalitions and their responsibilities (research question a) Several aspects of the development of a ‘joint programme’ (how partners are cooperating, purpose; clarity on developing a Theory of Change, role of monitoring and joint learning, relation to other IA principles) (research question b, c and e) Roles of ICCO staff and external advisors (research question a) Development Results of programmes (research question d)

4. Research questions a. program coalitions (quantitative) • number of (functional) programmes per [date ] • Thematic orientation of program coalitions • Regional division • Role ICCO Alliance in the PA • Funding modalities b. ESP 3 aspects: systemic change • Who are the members of the program coalitions? Are members stemming from the pool of existing ICCO Alliance partners; are members stemming from other civil society partners or other actors ( like government, knowledge institutions, private sector, etc) • In what ways and directions are program coalitions developing?

• Roles, responsibilities of the various members of the coalitions • to what extent does the use of the programmatic approach as methodology strengthen the capacity to result in sustainable forms of cooperation between different actors in Southern countries c. ESP 3 aspects: co-­‐creation • To what extent does the programmatic approach contribute to the ability of co-­‐creation of different stakeholders in civil society?

• Are there differences in the application of the Programmatic Approach in the 6 thematic fields and how do they ‘show’?

Subject:ToR Programmatic approach status:draft Author: Dieneke de Groot; Hettie Walters

2009 - 2012

7 date: April 2013

•

What are ‘institutional’ opportunities and constraints within the ICCO Alliance regarding the Programmatic Approach?

d. assessing progress in the programmes to date • What have been the most significant results of working with the programmatic approach to date? e. coherence with cross cutting principles • How are gender equality and human rights being integrated / taken up in programs?

5. Methodology The evaluation will be carried out in the form of a desk study (synthesis and analysis of existing documents, available studies and other relevant material within ICCO /Alliance), in combination with interviews/ conference calls and a survey among ICCO Alliance staff and selected partner organizations working in a programmatic way. The latter will be used for triangulation. In the analysis stage the various data and findings will be compared and analysed to answer the research questions.

Based on the analysis of the data gathered the evaluators will formulate a specific set of questions which still have to be answered and can only be answered by field work research. They develop a motivated proposal for a phase 2 of additional field work. The field work has to be designed to find answers on these questions by visiting a maximum of three of the 5 programmes studied in the action.

In the whole evaluation process the evaluators should be open to ‘surprises’’ and unintended effects (both positive and negative) of working with the Programmatic Approach.

Methods of data collection and analysis: • Secondary data review of the various reports on programmatic approach mentioned under point 10, bibliography • Conduct semi-­‐structured interviews with approx. 35 ICCO Alliance staff (group interviews with ICCO KerkinAktie specialists in Utrecht; staff of Edukans and Prisma), and in the ROs (by Skype). The selection should be made in such a way to be able to answer the evaluation questions b -­‐ d.

• Telephone interviews (by Skype) with approx. 10 programmatic coalitions and the actors involved in those coalitions (see PSO evaluation) in the South who are working in a programmatic way.

• Online anonymous survey for ICCO staff and partners in the South on working with and the results of the programmatic approach, and simple statistical analysis. and under the condition of a well substantiated proposal • Field research (interviews with various stakeholders and beneficiaries), focus group discussions with beneficiaries), network analysis and power analysis) to complement and consolidate findings from the other evaluation questions (especially on question d and e).

6. Deliverables A draft report in English; to be submitted December 31, 2013 latest.

A final report in English (max 25 pages), excluding annexes. The format of the report will be in electronic version and in hard copy (5 copies). The final report should be submitted within 7 days after receiving ICCO’s comments on the draft report.

7. Planning Part of the evaluation time Subject:ToR Programmatic approach status:draft Author: Dieneke de Groot; Hettie Walters

2009 - 2012

8 date: April 2013

Phase 1

Document study 5 days Group interview (5, with 5-­‐8 Alliance staff) 6 days Skype interviews RWO staff (16) 8 days Individual interviews (5, with RMs, R&D staff) 3 days Survey (including preparation, distribution and processing) 5 days Interviews with coalition-­‐ partner organizations (10) 5 days Report writing phase I 5 days

Subtotal 37 days

Phase 2

Field visit (to max. 3 programme coalitions, 5 days + travel) 21 days Report writing phase II + overall report 3 days Finalisation 2 day

Subtotal 25 days

Total 62 days

The evaluation report will be finalized by of 2013.

8. Profile evaluator -­‐ background in sociology, cultural anthropology, human geography, development studies or similar field -­‐ knowledge of working in a programmatic way (and ICCO’s interpretation of this concept) -­‐ experience with social survey methods (data collection, entry, analysis), semi-­‐structured interviews and focus group discussions -­‐ analytical skills -­‐ at least 5-­‐10 years’ experience working in the South -­‐ English, French, Spanish and possibly Portuguese language skills The possibility exists to carry out the evaluation in a small team of 2 evaluators.

9. Budget The evaluation PW is part of the budget set aside for PMEL. The maximum budget available for the evaluation is € 55.000 (VAT included).

Payments: The payment procedure is the following: 30% at acceptance 30% at presentation draft report 40% after receipt of approved final report and financial justification

10. Bibliography Will be available at the start of the evaluation.

Subject:ToR Programmatic approach status:draft Author: Dieneke de Groot; Hettie Walters

2009 - 2012

9 date: April 2013

Annex 2 – Workplan ICCO Evaluation Programmatic Approach

ICCO Evaluation Programmatic Approach WORK PLAN

THE PURPOSE AND OBJECTIVE OF THE EVALUATION In the words of the ToR for this evaluation, the purpose of this evaluation is to further deepen the understanding of the programmatic approach in practice and its contribution to the implementation of effective programs reaching their (change) objectives. The main focus is on critical aspects of the programmatic approach like ownership, stakeholder diversity and sustainability, and to grasp the factors related to these issues. This evaluation purposefully builds on the various (internal) studies done on the programmatic approach by giving a comprehensive external view.

The objective of the evaluation of the programmatic approach is to get insights into some more qualitative and some quantitative questions regarding the programmatic approach. The outcomes of the evaluation will help to further consolidate the approach for the remaining period of the business plan 2011 -­‐ 2015 and for policy development beyond 2015.

THE MAIN EVALUATION QUESTIONS For the ICCO Alliance, the most urgent questions for now are: With regards to the programmatic approach: what works well and what lags behind or doesn’t work with regards to the introduction and implementation of the approach and the learning thereon, and what factors contribute to this?

With regards to programmatic coalitions and programmes7: how have these programmatic coalitions and programmes taken shape, who are their stakeholders, and who has their ownership?

In dealing with these overall questions, the evaluators will consider four specific dimensions in their analysis: actors (who is involved), roles and responsibilities (who does what why), process (who does what when and how), and results (to what does all of it lead).

MORE SPECIFIC RESEARCH ISSUES During their research, the evaluators will treat all the following specific issues to enable appreciation of the abovementioned questions: a) Programme coalitions and programmes – initial sketch • Evolution of the number of (functional) programme coalitions and programmes over time; • Thematic orientation of programme coalitions and programmes; • Members of the programme coalitions (origin, type); • Geographical location of programme coalitions (regions and countries); • Funding modalities of the programme coalitions and programmes; • The instruments put in place to promote the emergence of programmatic coalitions and to manage the programmes.

During the preparatory desk study phase, the evaluators will further clarify and define the way in which they will work with the terms and concepts ‘programme coalition’, ‘programme’, ‘ functional’, ‘ themes’, ‘origin & type’, and ‘funding modalities’ mentioned above. They will do so based on the documents at hand and in close collaboration with ICCO staff concerned. b) Programme coalitions and programmes – more in detail (systemic change and co-­‐creation8) 7

Note: the phenomena ‘programme’ is added to the scope as it seems at times to be a starting point for coalitions as well as at times its outcome and means to impact Annex 2 – Workplan of the Evaluation Programmatic Approach

10

• • •

Evolution of the programme coalitions and variations thereof in various thematic and geographical settings; Roles and responsibilities of the various members of the coalitions and the ways in which these have evolved (including ICCO staff and external advisors); The sustainability of the cooperation within the programme coalitions.

c) The programmatic approach • The evolution of the programmatic approach, and the geographical and thematic variations therein.

• The effectiveness of the programmatic approach as a methodology to strengthen sustainable forms of cooperation between different actors in Southern countries (or: the degree to which the programmatic approach has contributed to the emergence of sustainable programme coalitions – see above); • The extent to which the programmatic approach contributes to the ability of co-­‐creation of different stakeholders in civil society; • The ‘institutional’ opportunities and constraints within the ICCO Alliance regarding the programmatic approach. d) Assessing progress in the implementation of the programmatic approach9 to date • The most significant results of working with the programmatic approach to date. e) Coherence with cross cutting principles • The ways in which and the extent to which gender equality and human rights are being integrated / taken up in programmes.

METHODOLOGY The evaluation will be executed during a number distinct phases: A document study phase during which the evaluators will analyse existing documents, available studies and other relevant material in order to get to grips with the issues at stake. It will generate the first, very rough overviews of the emergence and evolution of the various key-­‐concepts with regards to the programmatic approach. It will result amongst others in an unpolished timeline showing the evolution of the programmatic approach. Some short interactions are foreseen with key IA staff for clarification purposes. At the start-­‐up of this phase, the evaluators will draft a detailed ToR for a junior who will browse through all IA-­‐systems and contact relevant staff to generate initial tabular overviews with quantitative data with regards to programme coalitions and programmes.

The evaluators will guide the junior during the process.

These very first insights generated during this phase will also result in the formulation of a series of questions that will form the basis for: An interview and data-­‐gathering phase during which the evaluators will engage with: • 35 ICCO Alliance staff (semi structured group interviews with Dutch-­‐based thematic staff from ICCO KerkinAktie, Edukans and Prisma); • 16 ICCO Alliance staff from the regional offices (through semi-­‐structured Skype-­‐interviews); • Representatives of 10 programmatic coalitions (through semi-­‐structured Skype-­‐interviews); • Regional managers and R&D staff (though inquisitive conversations); • A broader group of ICCO Alliance staff and partners in the South on working with and the results of the programmatic approach (through a semi structured, anonymous online survey).

The outcomes of this phase will be used to adapt and further fine tune the insights gathered during the desk study phase and to add first and indicative appreciations of results generated through 8 9

The concepts systemic change and co-creation did not seem to cover the issues mentioned thereunder Note: the original research questions mentioned ‘programmes’ here

Annex 2 – Workplan of the Evaluation Programmatic Approach

11

working with the programmatic approach and programme coalitions. The first timeline will be fine-­‐ tuned and critical factors during the evolution of the programmatic approach will be highlighted.

A reflective and initial report writing phase during which the evaluators will analyse the undoubtedly varied picture that emerged from the inquisitive reading and listening phases and produce a synthesis capturing the key elements with regards to the evaluation questions and the underlying dimensions. During this phase, the evaluators will have at least one encounter with key ICCO staff to sharpen the synthesis as it emerges. As this synthesis might most probably be enriched and deepened by first-­‐hand, real life experiences and stories directly from field-­‐level, the evaluators will produce a detailed and substantiated proposal to finalise the evaluation of the programmatic approach with a field research.

A field research phase during which the evaluators will conduct interviews and focus group discussions with various stakeholders in programme coalitions and beneficiaries of programmes implemented, to complement and consolidate findings from the earlier evaluation phases.

A final reporting phase during which the evaluators will finalise the initial report and add insights gathered during the field research phase. During this phase, the evaluators will include a final iteration with key ICCO staff in order to incorporate their comments in the final document.

RESULTS The results of the evaluation will allow the ICCO Alliance to: • have deeper understanding in the evolution of the programmatic approach, both in relation to the emergence and functionality of the programmatic coalitions as well as in the way in which programme coalitions work towards reaching their development objectives through joint programmes; • get pointers for further adaptation and consolidation of the programmatic approach; • gain insight in the contribution of the programmatic approach to the realisation and implementation of the current strategic plan; • gain insight in the role the ICCO Alliance plays in strengthening or hindering the development of fruitful cooperative processes; and to • gain insight in the relation between the programmatic approach and the other crosscutting approaches : Gender and the Rights-­‐based approach.

TIME PLANNING AND BUDGET The initial report will be delivered before the end of July. The end report (including the outcomes of the field visits) will be delivered as soon as possible after the field visits before the end of December of this year.

The number of days presented in the table below deviate from those indicted in the ToR. The evaluators believe that for certain activities more time is required, e.g. document study in view of the number of documents provided; certain activities (group interviews) will partly involve the input of both the evaluators; while the coordination with IA key staff in both phases was initially not foreseen.

The junior indicated for the document study phase is not included in the budget. His/her assignment will most probably take between 5-­‐10 days, depending on the presence of readily available data and/or the challenges to further unearth these.

Annex 2 – Workplan of the Evaluation Programmatic Approach

12

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

ICCO Evaluation Programmatic Approach INITIAL SYNTHESIS OF FINDINGS

INTRODUCTION The evaluation of the programmatic approach as shaped and implemented by the ICCO Alliance (IA) is subdivided in two phases: • a phase during which the evaluators read and analyse all material available on the launch, emergence, and implementation of the programmatic approach, including material on the subsequent learning thereon, gather additional data in dialogue with key players in the ICCO Alliance, and share their first thoughts with them; • an optional phase of field visits to three countries to better understand and document the conditions and factors that contribute to the emergence of successful coalitions.

The online survey directed towards IA programme officers and representatives of coalitions that was initially foreseen to be undertaken during the first phase of the evaluation will be held during the month of August.

This paper presents the initial findings and some first recommendations based upon the first phase of the evaluation. The outcomes of the online survey will be included in the final report on this evaluation. The current paper also includes a first series of thoughts and ideas on the optional phase of field visits. As per explicit request, this paper will be crisp and brief to enable its practical use during the upcoming gatherings and strategic deliberations of the ICCO Alliance leadership. As a consequence, this paper does not present the extensive evidence-­‐base on which the evaluators have grounded their findings and recommendations and does not always follow the classical rules for evaluative studies. The evaluators are however available to provide for detailed explanations to and share their coded data-­‐base with all interested parties. Although all findings and recommendations are based upon the material provided and the conversations held, the evaluators are solely responsible for their formulation.

THE MAIN EVALUATION QUESTIONS In the words of the ToR for this evaluation, the purpose of this evaluation is to further deepen the understanding of the programmatic approach in practice and its contribution to the implementation of effective programmes reaching their (change) objectives. The main focus is on critical aspects of the programmatic approach like ownership, stakeholder diversity and sustainability, and to grasp the factors related to these issues. This evaluation purposefully builds on the various (internal) studies done on the programmatic approach as well as on all conversations held by giving a comprehensive external view.

The objective of the evaluation of the programmatic approach is to get insights into some more qualitative and some quantitative questions regarding the programmatic approach. The outcomes of the evaluation will help to further consolidate the approach for the remaining period of the business plan 2011 -­‐ 2015 and for policy development beyond 2015.

THE INTRODUCTION OF THE PROGRAMMATIC APPROACH AND THE LEARNING THEREON In introducing the programmatic approach, the IA has followed a ‘learning by doing’ approach. Since 2006 a rich series of documents has seen the light in which the ins and outs of ‘the programmatic way of working’ have been indicated and explained. Parallel to these, and often in strong interaction with these, the IA has invested heavily in organising all kinds of learning trajectories, appreciations,

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

13

action researches and training exercises to guide and support the IA staff as well as partner organisations in further defining and working with the programmatic approach. There is an abundance of papers, notes, workshop debriefings and syntheses indicating the evolution of thoughts and experiences on introducing and working with the programmatic approach. (See the Annex es B and C to this report for a full overview of documents and peoples consulted.)

Findings 1.

The choice for a ‘learning by doing’ approach was necessary as the programmatic approach was described in bold and general terms only at its launch and as the IA staff and partners had no prior experience in working in this way.

2.

The ‘learning by doing’ approach seems furthermore justified, as working in a programmatic way, through its very nature, simply requires such an approach.

3.

All documentation on the different learning processes present a clear and very rich picture on the factors contributing to the approach’s success as well as to the bottlenecks it faces.

4.

Although the documentation contained quite some critical remarks on the introduction of the programmatic approach right from the start, the management responses to these seem much less obvious and clear. This justifies the finding that learning has definitely taken place, but also that most often the learning loops did not seem to have been closed.

The introduction of the programmatic approach went hand in hand with the ‘co-­‐responsibility’ process (i.e. bringing the design and decision-­‐making on the processes of change closer to its partners and context) and the ‘decentralisation’ process of the operations to the regions (meant to better execute different roles indicated in the IA business plans). These three major innovations have become known as the PROCODE process.

It is clear to the evaluators that these three major change processes are highly interrelated and that their advantages seem obvious (including partners in decision-­‐making, better context knowledge and closer interaction between IA staff and IA partners). All documentation however reveals that the parallel launch of these three major change processes has complicated the introduction of the programmatic approach and might even have set it off in other directions than initially intended.

Findings 5.

In the eyes of many partner organisations (and IA staff) the introduction of the programmatic approach was linked to the decentralisation process and associated directly with the new thematic focus of the IA indicated in various business plans. The changes were perceived to be all directed by ICCO, changing the terms of the cooperation with its partners. This was in many cases not perceived as ICCO intending to share power, but as ICCO taking the reigns of the cooperation stronger in its own hands.

6.

Introducing three major changes at the same time has certainly put severe stress on the IA staff and systems. Many a document indicates that IA staff simply did not and/or could not invest sufficient time in the often delicate processes required by the programmatic approach and that the PMEL systems were not fully geared towards supporting the introduction of the programmatic way of working.

Next to the ‘learning by doing’ approach and the launch of three major changes at the same time, a third factor has had a heavy bearing on the ways in which the programmatic approach has actually materialised in the IA practise.

Right from its inception in 2006, all notes and guidance on the programmatic approach have stressed the significance of local ownership of the problems and the solutions thereto, and of the importance of tuning interventions to the local context, to the dynamics between local stakeholders and to their agenda’s, ambitions, roles and opportunities. Flexibility in timing and planning and true local ownership over the process are key ingredients of successful implementation of the programmatic approach.

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

14

The programmatic approach was however not launched in splendid isolation, but as a way of working to implement the IA business plans 2006-­‐2010 and 2011-­‐2015. These business plans present specific thematic focus areas, and detailed objectives and results to be achieved.

Findings 7.

All documents and experiences reflect the true balancing act between learning by doing with an approach that requires flexibility and a focus on local actors and dynamics on the one hand and implementing the IA business plans with predefined themes, fixed result frameworks and procedure driven planning and reporting formats on the other.

8.

As a matter of course, the balance has often tipped in favour of implementing the IA business plans, resulting in what became known as ‘programme-­‐building’ with the IA in the driver’s seat of programmes and coalitions.

CONCEPTUAL CLARITY ON KEY CONCEPTS Appreciating ‘what works well and what lags behind or doesn’t work with regards to the introduction and implementation of the programmatic approach and the learning thereon’ (one of the evaluation questions), requires clarity on the key concepts of the approach and an appreciation of the degree to which this clarity is indeed shared amongst the rank and file of the IA staff and its partners. In other words: “What is the programmatic approach?” And: “Do all concerned understand it in the same way?” To this end, the evaluators have first analysed all relevant documents and established a timeline indicating the evolution of the various key concepts with regards to the programmatic approach. They have secondly added on this timeline various, often critical, remarks and observations on the concepts that emerged from earlier evaluations and the different learning histories, appreciations, action researches that the IA undertook during the 2007-­‐2012 period. A summarised overview of the timeline is presented in Annex A to this report.

Findings 9.

The document review and interviews so far have revealed that there is no common understanding amongst IA staff and partners of what is to be understood by the programmatic approach. Descriptions vary from ‘cherishing and facilitating local ownership of development’, to ‘a programme‘, to ‘a blue-­‐printed approach with the IA in the lead’, to ‘a quest for the holy grail’, to ‘a belief-­‐system’.

10. The recent studies and the MTR have revealed such a broad variety of programmes and coalitions, of typologies thereof, of organisational models, of governance structures, of contracting arrangements, and of funding modalities that it is hard to distinguish a common denominator under which these can all be labelled ‘fruits of working with the programmatic approach’.

11. There are a number of key-­‐traits of the current IA way of working that are cherished by most people interviewed and that these wish to further exploit and take on board in the formulation of future strategy and policy currently underway (see below).

12. Document analysis did not reveal any specific attention to, or development and introduction of practical instruments on gender equality and human rights.

As the timeline clearly indicates, the key-­‐concepts on the programmatic approach have evolved over time. The description most consistently in use is the one introduced in 2010: The programmatic approach is defined by ICCO as a process that leads to organisations working together based on a joint analysis, shared vision and objectives and clear perspective on the results of the cooperation. In such a process all actors can do different things, work at various levels and use their own strengths for the common purpose and objectives, as well as share some activities and in particular share and participate in the linking and learning processes. The programmatic approach is an approach that does not only address single problems but aims at change of systems.

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

15

The programmatic approach is described as a process. In the earlier years during which the most programmes and coalitions were shaped, descriptions of the approach however most often referred to ‘programmes’, ‘interventions’, ‘combinations of the four IA roles with the business plan themes’, and ‘stakeholders collaborating in a programme’. The most recent paper in 2013 on the programmatic approach again linked the approach to the realisation of results in ‘identified thematic domains’, without clarifying how that was to relate to the alleged key importance of ‘local ownership’.

Findings 13. Comments on the apparent lack of clarity and perceived inconsistencies on the key-­‐concepts of the programmatic approach have been circulating right from the launch of the approach in 2006. Despite massive efforts, this lack of clarity continues to date.

14. The absence of quality standards on ‘good programmatic practice’ has not contributed to creating the necessary conceptual clarity.

Understanding why this ‘experienced confusion’ and ‘perceived inconsistency’ has been so persistent throughout the entire 2006-­‐2013 period under consideration remains one of the big challenges of the current evaluation.

The simplest explanation seems to be that the confusion is built right into the name ‘programmatic approach’ and its positioning as a way of working to achieve the ambitious results in the six programmes that are indicated in the IA business plans. Simply put: the bulk of the IA staff seems to have understood the approach as a means to shift from a ‘one partner-­‐one project-­‐one contract’ way of working, to a ‘coalition and programme building’ approach, in which they have taken the lead and indeed started to create coalitions and programmes on the six IA themes (or programmes!).

Findings 15. IA guidance on the programmatic approach has not sufficiently clarified the major strategic question whether the IA core-­‐business is about ‘brokering locally owned solutions to locally owned problems within local settings and dynamics following local agendas and rhythms’, or about ‘facilitating multi-­‐stakeholder processes to support joint action to realise the results on programmes as indicated in the IA business plans’.

16. The apparent lack of a common and shared understanding on the key concepts of the programmatic approach has given room to the organisations participating in the IA, to the regional managers and to the thematic departments ‘to fill in the blanks themselves’ which has not helped in creating a consistent story on the IA way of working and an alluring and unambiguous corporate profile.

THE PROGRAMMATIC APPROACH AS ‘EFFECTIVE SELLING POINT’ During discussions with the IA leadership (directors and regional managers) it became clear that the IA way of working is hardly ever explained to outsiders (or sold to them for that matter) using the label ‘programmatic approach’.

In this context is seems fair to say that the programmatic approach is neither new nor innovative.

The approach was introduced in development cooperation in the eighties and nineties, under the label ‘process approach’ and also ‘programmatic approach’. The approach lost traction, as many a donor appeared not to be able to handle the long-­‐term, open-­‐ended and flexible nature of the interventions that resulted from working with it (just to mention one of the reasons).

A number of elements of the programmatic approach are however much cherished by most people interviewed, and indeed often mentioned in explaining the IA way of working to partners and outsiders alike. These elements are considered to constitute the ‘pearls of the IA practice’ that are most definitely to find their way into the future formulations of the IA way of working:

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

16

•

• •

• • •

for change processes to be relevant, they are to be grounded in a thorough and continued understanding of the ever-­‐changing context, issues at stake, power relations amongst the major actors and the dynamics on how these actors relate to one another and to the systems of which they are part; for change to happen, working at multiple levels with multiple stakeholders is necessary; for stakeholders to indeed join forces effectively and enter into some form of joint and emergent action, a shared vision on the change and transformation to be promoted and about how to go about that (in other words a mutual consensus on a ‘theory of change’) seems crucial; for change to be sustainable, is seems crucial to address ‘the big issues’, the systems that generate poverty and injustice; for change to be effective, learning on what happens, on what works and what does not and consequently flexibly adapting one’s set of activities is key; relations amongst the cooperating parties are based upon trust and mutual accountability.

In a simple picture the main results of the IA way of working: joint analysis, shared vision and a clear perspectives on the results to be achieved organisation (NGO) action

organisation (govt.) learning big issue organisation (private)

organisation (FBO) cooperation organisation 'coalition'

Findings 17. The ‘ideal type’ IA programmatic approach is more about ‘facilitating local change processes’ than about ’formulating and implementing log-­‐frame type programmes’.

18. The major challenge seems to be on how to couple the ‘local ownership over development processes’ as advocated by the programmatic approach, with the evident own institutional interests and development agenda of the IA. It is remarkable to note that these key-­‐traits very much resemble the original IA way of working as presented in the 2006 business plan (see Annex A). Then again, why did this way of working not materialise in a consistent way everywhere in the IA practice?

Apart from the argument and explanation introduced above that led to programme and coalition building, a number of other factors have contributed to an all but full-­‐fledged introduction of the programmatic approach. These are linked to the IA roles and the IA management instruments.

ROLES OF IA STAFF As indicated in all major policy documents and business plans, staff of the IA in general and the programme officers more in particular are to play a role as strategic funder, broker (knowledge and relations), supporter of capacity development (of the cooperative initiatives and of its partner organisations) and acts as lobbyist and advocate for change in the global and the Dutch polity.

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

17

Of these four roles, the brokering role and the support to capacity development role are of key importance to a successful application of the programmatic approach.

Findings 19. The various documented experiences of programme officers in working with the various roles indicate a substantial tension between the funder role one the one hand and the broker and capacity builder role on the other. Experience apparently shows that the broker and capacity builder roles are compromised by the fact that the same institution that brokers and strengthens capacity is also the one that is willing, under certain conditions, to fund ‘those brokered and capacitated’.

20. Brokering as well as supporting capacity development are time-­‐consuming and delicate processes that require specific skillsets. Despite the massive investment in trainings, not all IA staff appears to have the necessary competencies, and those who do often have too little time available to do a proper job on the two roles. Recent studies have indicated that the bulk of time available (± 80%) is invested in ‘dossier’ management (part of the funder role?).

21. The IA managers and leaders appear not to steer on a balanced application of the various roles (nor on a successful application of the programmatic approach for that matter). Appreciating the quality of work is done on indicators like number of contracts signed and managed, adherence to financial, administrative and reporting procedures, and amount of financial resources mobilised.

In many a case, external consultants are hired to play the broker role and the supporter of capacity development role. The documentation indicates that most partners are very satisfied with the quality of work done by these external parties.

Findings 22. Two of the four roles in the core business of the IA staff seem to deliver the best results (both quality and time wise) when executed by external consultants.

Recently a fifth role has been added to the four roles mentioned here above: the fundraiser role.

Explicit attention for this role is highly understandable in view of the substantial decrease of central level subsidy from Dutch Government to the IA that is foreseeable in the not too distant future.

Findings 23. From the various conversations it did not become clear if the fundraising role concerns mainly fundraising for the IA, and, if so, whether or not that role is compatible with fundraising to enhance the financial sustainability and autonomy of partners and coalitions.

During the various exchanges with IA leadership, a sixth role surfaced that will most definitely have its bearings on the IA way of working: the co-­‐implementer role. In this role, the IA becomes an active stakeholder in development programmes and activities and will bear more direct responsibilities for the production of development outputs and outcomes. The driving force behind wanting to play this role seems to be that it might increase the IA opportunities for fundraising.

Findings 24. The fundraiser and co-­‐implementer roles might result in a direct competition between the IA, its partners and other local actors on the decreasing ‘development funds market’.

25. The co-­‐implementer role bears the risk of redirecting energy and attention from ‘supporting local development processes’ to ‘formulating bankable proposals that serve the donor’s agenda’.

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

18

MANAGEMENT TOOLS FOR THE PROGRAMMATIC APPROACH The IA has introduced management information systems (PME tools and procedures, financial and administrative systems and formats) that are especially geared towards the management of the implementation of the IA business plans. Regular information is gathered on the status of contractual and financial obligations on partners and the IA programmes and on the indicators for success as formulated in the monitoring protocol agreed upon with the Dutch Ministry for Development Cooperation.

Findings 26. The current systems do not provide for valid and regular management information on numbers and qualities (and the evolution thereof) of programmes and coalitions. Timely introduction of the P-­‐Scan would have improved the current lack of quantitative and qualitative management information on entities crucial to monitor progress on the implementation of the programmatic approach.

27. The current push for one-­‐year contracts does not help in creating the right conditions for the further emergence of multi-­‐stakeholder dynamics and action.

INITIAL RECOMMENDATIONS The evaluators have had the honour of having a brief look in the kitchen of the IA. They have encountered and exchanged ideas with very dedicated and highly professional staff and leadership.

They have engaged with the team responsible for the shaping of the programmatic approach and the training and learning thereon. They have deliberated with the staff responsible for thematic quality and coherence and with the men and women that have to translate central ambitions to the day-­‐to-­‐ day practice in field reality. And they have conversed with the four directors of the IA that are there not only to inspire and motivate, but at the end of the day also to account for the results achieved.

The challenges are vast: realising the IA ambitions and preparing for the future in a rapidly changing development sector.

The evaluators’ findings and recommendations are to be seen in this light: observations and thoughts of outsiders that fully realise the different roles that everyone plays in a large organisation as the IA and that fully recognise that under pressure many a thing becomes liquid. Observations and thoughts that are meant to support the IA in reflecting on strategic issues and on future positioning.

In summary, the introduction of the programmatic approach in the IA has not reached its full potential as a result of: • an apparent lack of a common and widely shared understanding on the meaning of the key-­‐ concepts of the approach; • a serious and unresolved strategic tension between implementing an ‘ideal type programmatic approach’ and realising the IA business plan results; or to rephrase: • the fact that at conceptual level the IA has not been able to properly balance ‘local ownership’ with the alliance’s own interests and development agenda; • the difficulties (both strategically as well as quality and time wise) in fully realising two of the four/six IA roles (brokering and support to capacity development); • the fact that the IA management tools and HR systems have not been geared towards information on and steering on a successful implementation of the programmatic approach.

The IA leadership is currently engaged in a ‘multi annual strategic planning’ process. During this process, the IA leadership will most definitely spend some quality time together to thoroughly deal with a number of key issues and questions with regards to the future IA positioning and issues that all have to do with the question who you are and what you want to be: • Are the IA vision and mission statements still up to date?

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

19

• • •

•

What are the defining elements or key traits of the IA as a Dutch civil society organisation with its roots in the protestant community?

Are the ways and means in which the IA members relate to and communicate with their constituencies still up to date?

What is the added value and specific niche of the IA in development cooperation? In what does the IA distinguish itself from the numerous other INGO active in (implementing) development (programmes)?

What is the most suitable business model to bankroll the above? Do we follow the money or do we mobilise money for what we want to do?

Depending on the outcome of the conversations mentioned above, it is highly recommendable to: 1.

Deal with the two major strategic issues mentioned in this report: a. how to couple the ‘local ownership over development processes’ with the evident own institutional interests and development agenda of the IA? • define ‘local ownership’ and indicate what the IA considers to be within the boundaries of ‘local ownership’; • define ‘partnership’ and indicate what the IA considers to be the qualities of ‘true partnership’; • define and make explicit the institutional interest of the IA in terms of branding, national, regional and global presence, minimum staff and budget levels; • define and make explicit the IA development agenda in terms of themes and/or processes to be supported and in terms of types of results to be achieved. b. is the IA core-­‐business about ‘brokering locally owned solutions to locally owned problems within local settings and dynamics following local agendas and rhythms’, or about ‘facilitating multi-­‐stakeholder processes to support joint action to realise results on programmes as indicated in the IA business plans’?

2.

Rephrase the term ‘programmatic approach’ using internationally accepted concepts and terms.

In doing so, use could be made of the ‘pearls of the IA practice’ mentioned in this report and to firmly anchor these pearls in a widely shared understanding of ‘local ownership’. The ensemble of these ‘pearls’ could be simply labelled as the ‘IA way of working’. Clarifying furthermore once and for all the concepts ‘local ownership’, ‘programme’ and ‘coalition’ and how these are to relate to the IA themes and result frameworks is of key importance.

3.

Depending of the outcomes of the conversations indicated above, it seems advisable to redefine the IA roles. The current six roles position the IA as ‘a jack of all trades’, in practice having difficulties to deliver high quality services on a number of them. The suggestion here is to focus on the roles crucial for the IA core-­‐business. And that could be either in for example the co-­‐ funding business (the IA channelling funds from its constituencies in Europe to partner organisations in the South), in the brokering business (the IA brokering local solutions to local problems), or in the co-­‐implementation business (the IA as one of the key actors in managing the implementation of development programmes). In focussing on the roles crucial for the IA core-­‐business, it is important to realise that the ‘pearls of the IA way of working’ are all specifically designed for and do indeed all favour the IA in broker and capacity developer roles.

4.

Systems, tools and procedures are to support the IA in implementing its core-­‐business, in maintaining quality standards and control and in assuring accountability over the operations. In view of the above, it is highly advisable to adapt the IA management information systems, the PM&E systems and the HR systems to fully support the IA way of working. Formulating quality standards for the IA way of working seems a prerequisite for this.

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

20

5.

For the second phase of the current evaluation assignment, it is advised to undertake three fieldtrips to coalitions (in the broadest sense of the word) that are to be considered interesting and successful. The main task for the evaluators would be to engage with all concerned (especially the participating organisations, but also IA staff) and to: a. identify why these coalitions are considered successful; b. describe the process of their genesis; c. describe what they do, why they do it and how they go about in doing it; d. analyse the various factors that have contributed to their success (external environment, nature and composition of cooperating partners, type and quality of support provided, governance arrangements, planning and learning dynamics, funding flows and mechanisms, personal qualities, happenstance); e. chart the factors that might influence the future success and the viability and sustainability of the coalition; f. document the above in crisp, brief and clear papers highlighting best practices in the IA way of working.

It would be most interesting to visit three coalitions that have not already participated in the appreciations and action researches or have been otherwise documented.

September 2013, Verona Groverman & Kees Zevenbergen

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

21

Annex A. Timeline Evolution Key-­‐Concepts Programmatic Approach Statements on key-­‐concepts

Critical remarks (all from documentation) Business Plan 2006-­‐2010 2006 ICCO Alliance way of working: • context analyses country & theme • stakeholder & drivers of change analysis • stakeholders collaborate in a programme and strive for change in power relations • stakeholder agree upon monitoring, learning and evaluation Start Paper Programmatic Cooperation

Start Paper Programmatic Cooperation The programmatic way of working implies The concepts ‘programme’, ‘project’, ‘theme’ combining the four roles (strategic funding, and ‘programmatic approach’ are often used brokering, capacity development and interchangeably, which does not help in advocacy & lobby) within the business plan clarifying matters. themes.

A programme is a consistent combination of the four roles in a specific area.

The Programmatic Approach in the ICCO 2007 Programmatic Way of Working Alliance • give more time for introduction By a programmatic approach we mean all of • improve guidance on creation of the interventions during a certain period of programmes time and within a particular geographical • improve linkage of PRO-­‐DE-­‐CO area. Organisations with different areas of • increase communication on PA expertise and experience participate in such a • improve learning on PA programme. Together they try to reach agreement on its vision, objective, implementation strategy and the roles played by the various organisations.

Programmatic Approach at Work 2008 Programmatic Approach at Work The word “programme” is often confusing as • The programmatic approach influences the it can mean many things and therefore can dynamics of change in a society be understood in many different ways.

• Local ownership is a crucial and decisive The separate programmatic departments in element!

ICCO/KIA are not so good for initiating and • Different actors join forces and decide to support to multi-­‐thematic programmes. engage into something new • The programmatic approach changes the way in which partners relate to their society • The ICCO Alliance supports the initiation of southern programme coalitions The Programmatic Approach 2009 The Programmatic Approach The main organisational strategy for Experience shows that the process of coming cooperation with partners that ICCO (and to a to a joint vision and objectives and the different degree some of it’s alliance process of motivating actors to become partners) have adopted is the programmatic involved in a programmatic cooperation approach. Through the programmatic process is lengthy and not easy. approach the ICCO-­‐Alliance seeks to promote The ICCO Alliance does not intend to “build” and support multi stakeholder co-­‐operation ICCO Alliance programmes but intends to processes needed for sustainable reduction promote and support programmatic

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

22

of poverty and injustice.

cooperation between relevant actors in a context.

Appreciating the Programmatic Approach “My overall assessment is that for southern partners to really take ownership of these emergent ‘organisations of organisations’ will take many years and very high quality support and accompaniment by those who have initiated them” (JamesTaylor) Evaluative Study ICCO Programmatic Approach There evolved a clearer notion of a difference between a ‘programme’ (content) and the Programmatic Approach (process). While in some cases there are programmes that are just administrative clusters of activities and contracts, others are the result of a process of partners defining some joint strategies with actions. All the variations carry the title ‘programme’, which therefore has made the concept -­‐ as a common denominator -­‐ rather useless.

Evaluative Study ICCO Programmatic

Approach Under a programmatic approach different organisations agree to work towards one concrete common goal. They work together on the basis of a common vision, core values, goal and a common strategy. Though every actor may play a different role using its own strength and networks, the end result of their collective intervention reaches a greater impact in society.

The four key principles of the Programmatic Approach are therefore: 1) working towards a common goal; 2) local ownership of the programme and its governance; 3) multi-­‐ stakeholder involvement; and 4) added value of the different actors involved.

Synthesis Paper 2010 2010 Synthesis Paper 2010 The Programmatic Approach is defined by The Programmatic Approach is seen as an ICCO as follows: A process that leads to element of the decentralisation process and organisations working together based on a associated to the new thematic focus of joint analysis, shared vision and objectives ICCO. and clear perspective on the results of the ICCO took the role of initiator of cooperation. In such a process all actors can programmatic cooperation. do different things, work at various levels and This has led to a great variety of initiatives use their own strengths for the common under the heading ‘programmatic approach’ purpose and objectives, as well as share or ‘programme building’. some activities and in particular share and participate in the linking and learning processes. The programmatic approach is an approach that does not only address single problems but aims at change of systems.

Synthesis Paper 2011 2011 The current practice of the Programmatic Approach in ICCO shows a wide variety of programmes with different characteristics on the content (type of issue, scope, level), the composition (partners, non-­‐partners, networks), role of ICCO during the process, funding and governance. Various models for structure and governance have been identified.

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

23

Strategy Paper Effective Selling Points To realise its mission and to collaborate with different actors, a programmatic approach is basic to the work of the IA. Programmatic working is defined by the IA as a process leading to multiple stakeholders working together on joint analyses of given challenges, shared visions and clear perspectives and purposes on the results of the collaboration. The IA believes that bringing together joint and diverse perspectives is the only way to alter underlying systems in society.

Guidance Note for the Programmatic Approach The ICCO Alliance’s Programmatic Approach is an approach that is about emerging forms of organisations that come together to collaborate. These organisational forms are known by different names such as coalitions, alliances, networks, partnerships, joint ventures or federations.

In this paper we will use the term coalition

Appreciating the Programmatic Approach -­‐ debriefing note The danger of PA becoming too controlled, linear and driven by predefined results should be avoided and continuously monitored.

The changes within the organisation have increased administrative demands on the programme staff.

As in the past lack of clarity, communication, coaching and training on PA has caused some uncertainty and insecurity, the Global Office could make a public statement recommitting to PA and programme cooperation. Clarity on the approach will help the partners to see the added value of cooperation and instil motivation among the ICCO staff on the ground.

Appreciating the Programmatic Approach -­‐ workshop report Initially there was no clear-­‐cut guideline for PA, which caused some insecurity and uncertainty among the partners, and ROs/POs. Communications about the PA was neither continuous nor coherent. Different interpretations of PA caused differential understanding among the partners and POs.

2012 Guidance Note for the Programmatic Approach In the ICCO Alliance we have initially called them program coalitions or even shorter: programmes. This last term is however confusing because it is also used for the ICCO Alliance policy level, for a set of objectives, results and activities (projects) related to a thematic domain and for the cooperation between stakeholders on a problematic.

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

24

for the associative form of organisations working together for the realisation of a joint purpose.

Synthesis Paper 2013 In 2012 emphasis in the support of the implementation of the Programmatic Approach was on strengthening the capacity of the actors involved to cooperate within a Programmatic Approach so that the envisaged and joint development results are being realised within the identified thematic domains.

Working Better Together The different roles of ICCO, i.e. being a broker, capacity developer and funder, often created tension between ICCO and the coalition. The inherent ambiguity that is the consequence of an international donor organisation starting coalition initiatives, while at the same time aspiring for these coalitions to become independent leads to several challenges. These are related to ownership, coherence, clarity on complementarity and roles, and the relation with ICCO. In particular the funding role of ICCO creates an unequal power relation with the coalitions.

2013 Synthesis Paper 2013 Problematic issues on: • maturity of coalitions • learning within coalitions • brokering and facilitation • ownership • funding and fundraising • governance of coalitions

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

25

Annex B. Documents reviewed (mostly chronological)

DOCUMENTS RELATED TO THE DEVELOPMENT OF THE PROGRAMMATIC APPROACH -­‐ HD. 12 Maart 2007. Programmatisch werken in de ICCO Alliantie. Opzet en achtergronden -­‐ No author. Spring 2007 (22-­‐6-­‐2007 last saved). Programmatic Approach of the ICCO Alliance (ppt) -­‐ Derksen, Harry. 4 juni 2007. Voorstel 1. Programmatisch Werken -­‐ Klarenbeek, Mildred & Ooijens, Machteld 2007? Bijlage III. Systematisering van de ervaringen met programmatisch werken in de Ruta del Sol, een onderwijsprogramma in Peru -­‐ Derksen, Harry. 14-­‐4-­‐ 2008. Programmatic Approach at work. A second update spring 2008 -­‐ Walters, Hettie. 14-­‐5-­‐2009. The Programmatic Approach -­‐ State of the Art PA Aug 2009 v2 -­‐ Wortel, Erica & Jouwert van Geene. December 2009. Consolidating Experiences of ICCO’s Programmatic Approach, An evaluative study of the Programmatic Approach of the ICCO Alliance -­‐ No author. March 19, 2010. How the U Process can be used to strengthen a Programmatic approach to Development Cooperation. An exploratory conversation -­‐ Walters, Hettie. 28-­‐3-­‐2010. Synthesis paper : Findings and recommendations gained from the Evaluative study and the Appreciating the Programmatic Approach processes -­‐ Walters, Hettie. December 2010, Proposal for the prolongation of the Learning-­‐working Path (LWP) of the ICCO (Alliance) Jan 2011-­‐ Dec 2012 -­‐ Walters, Hettie. 31.1.2011. Synthesis paper : Findings and conclusions from the Evaluative study and the Appreciating the Programmatic Approach processes.

-­‐ No author. 3.2.2011. Appreciating the programmatic approach. How far have we come? Where do we need to go?

-­‐ Walters, Hettie. 2011. Guidance note for the Programmatic Approach of the ICCO Alliance -­‐ Walters, Hettie. 11-­‐4-­‐2012. Monitoring and progress meeting report based on the Planning Framework -­‐ ICCO. 170912. PROGRAMMATIC COOPERATION SCAN (P-­‐SCAN) -­‐ No author. May 2013? Synthesis of observations and feedback to reporting about the progress with the implementation of the Programmatic Approach in 2012 -­‐ Walters, Hettie. 8-­‐5-­‐2013. Prog. Approach rapportages with comments -­‐ P&D, June 2013. Synthesis of observations and feedback to reporting about the progress with the implementation of the Programmatic Approach in 2012

EVALUATIONS AND APPRECIATIONS (by type of evaluation/appreciation, chronological) -­‐ Pronk, Hester & Hettie Walters, 2009. ICCO’s Implementation of the Programmatic Approach: What can Learning Histories Tell Us? -­‐ Ramírez Eduardo, 2009. Assessment Report for the Paraguay Organic Programme -­‐ Taylor, James. November 2009. Appreciating the programmatic approach: A systematisation of experience. Report of the learning from ICCO / Kerk in Actie Conflict Transformation Programme Uganda -­‐ Taylor, James. January 2011. Appreciating the programmatic approach: A systematisation of experience. Second Report of the learning from ICCO / Kerk in Actie. Conflict Transformation Programme Uganda -­‐ CB =?? 20.9.2010. Report for ICCO: draft conclusions and recommendations -­‐ Ramírez, Eduardo November 2010. Report on Second Round of Interviews – Paraguay Orgánico -­‐ Bruinsma, Domien. January 2011. Report on the 2010 Appreciation process experiences within the West African Organic & Fair Trade Cotton programme -­‐ Context, international cooperation. February 28, 2011. Appreciating the Programmatic Approach; a systematisation of experiences. Report of the Workshop January 31 – February 4, 2011, Utrecht

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

26

-­‐ No author. 4.3.2011. Appreciating Programmatic Approach: Systematisation Of Experiences.

Debriefing Note From The Workshop Held On 31 January To 4 February 2011 At Utrecht -­‐ Bandyopadhyay Kaustuv K . February 2011? Bangladesh Water and Food Security Partnership Programme (BWFSP). Appreciating Programmatic Approach of ICCO Alliance, the Netherlands -­‐ Caubergs, Lisette. Juin 2011. Rapport d’appréciation de « l’Approche Programmatique» : systématisation de l’expérience pour la période 09/09 – 12/10 -­‐ Karssemeijer, Ward. 14-­‐6-­‐2012. End report Junior Action Researcher Burkina Fasso -­‐ Mulder, Hanske & Kees-­‐Jan Mulder, 15-­‐06-­‐2012. Final Report. Recherche et renforcement des champs de collaboration entre les membres entités de la coalition LIFE -­‐ Jong, Elja de. June 2012. End Report Junior Action Researcher. Programmatic Approach – Agro ecological Consortium Peru -­‐ Punt, Wievenlien. June 2012. Final Report Action Research Nepal Coalition on Food Security and Water -­‐ Kruiter, Ingrid. 14 June 2012. Final report Action Research Programmatic Approach Youth and Security Programme Central America (Programa de Seguridad Juvenil Centroamérica) -­‐ Jong, Elja de, Ward Karssemeijer, Ingrid Kuiter, Hanske Mulder, Kees-­‐Jan Mulder, Wievenlien Punt. July 2012. Working better Together. Action Research: The Programmatic Approach from the Perspective of the Actors Involved.

-­‐ No authors. 2013. Evaluation of PSO Learning Trajectories Jan 2011 – Dec 2012 -­‐ No author. May 2013? Results for: Survey for Evaluation of the Learning and Working Process activities 2011-­‐2012 TRAINING REPORTS (by type of training, chronological) -­‐ General Programme Regional Facilitators ICCO. 16 – 20 November 2009. Guest House ICCO -­‐ Report of the workshop on the Programmatic Approach for staff of ICCO Alliance. Programmatic way of working – Programmatic cooperation – Multi Stakeholder processes. Block 1 (of 3) 17-­‐18 February 2011 -­‐ Report of the workshop on the Programmatic Approach for staff of ICCO Alliance. Programmatic way of working – Programmatic cooperation – Multi Stakeholder processes. Block 2 (of 3) – 17-­‐18 March 2011 -­‐ Report of the workshop on the Programmatic Approach for staff of ICCO Alliance. Programmatic way of working – Programmatic cooperation – Multi Stakeholder processes. Block 3 (of 3) – 13-­‐14 April 2011 -­‐ Report of the workshop on the Programmatic Approach for regional facilitators of ICCO Alliance.

Programmatic way of working – Programmatic cooperation – Multi Stakeholder processes.

Utrecht, 8 -­‐ 12 November 2010 -­‐ CDI. May 2011. A series of training on methods and approaches for a Programmatic Approach within ICCO Alliance. A proposal for the ICCO Alliance -­‐ Walters, Hettie. Report of the workshop on the Programmatic Approach for regional facilitators of ICCO Alliance. Programmatic way of working – Programmatic cooperation – Multi Stakeholder processes. Wageningen, 10 -­‐ 14 October 2011 -­‐ Capacitación de Facilitatadores Alianza ICCO – Enfoque Programático. Lima (Perú), 7 – 11 Noviembre 2011 -­‐ Walters, Hettie. Report of the workshop on the Programmatic Approach for regional facilitators of ICCO Alliance. Utrecht, 16 – 20 November 2011 -­‐ Brouwer, Herman, Hettie Walters, Dete Aliah. Programme Overview ICCO Alliance Programmatic Approach. Training of Trainers Sanur Bali, Indonesia 27 February – 2 March 2012 -­‐ Report of the training on the Programmatic Approach of ICCO Alliance. Programmatic way of working -­‐ Programmatic cooperation -­‐ Multi Stakeholder Processes. Bali. 27 February – 2 March 2012 -­‐ Report of the training on the Programmatic Approach of ICCO Alliance. Programmatic way of working -­‐ Programmatic cooperation -­‐ Multi Stakeholder Processes. Kampala, 23 -­‐27 April 2012

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

27

-­‐ No Author. Induction process for new staff of RWO’s 1-­‐1-­‐2009 to1-­‐1-­‐2010 ADDITIONAL DOCUMENTS REVIEWED (by focus/item, chronological) -­‐ ICCO Alliantie. April 2006. Bedrijfsplan 2007 -­‐ 2010 -­‐ Derksen, Harry. 16 december 2008. Opzet jaarverslag 2008 -­‐ Ooijens, Machteld. 18 december 2009. Opzet jaarverslag 2009 -­‐ ICCO Alliance. Grant Application Phase 1 MFS II -­‐ ICCO Alliance. Grant Application MFS II Phase 2. From Aid to Entrepreneurship -­‐ ICCO Alliance. 27 January 2011. Framework letter Reporting & Planning 2010 / 2011 -­‐ ICCO Alliance. 27 January 2012. Framework letter Reporting & Planning 2011 / 2012 -­‐ ICCO Alliance. 28 November 2012. Framework letter Reporting 2013 -­‐ Context, international cooperation & Strategic Connections. June 10, 2012. Evaluation of the ProCoDe Pilot in Central and East African Region. Draft report & Annexes -­‐ Phlix, Geert. 2013 MTR ICCO Programme 2011-­‐2015. Assessment of the progress made and results reached in the period 2011-­‐2012. Draft Reports Southern Africa, Western Africa, Central & Eastern Africa, Central and South Asia, South East Asia, Central America, Latin America -­‐ No author. 7.6.2013. Strategy Paper Effective Selling Points ICCO/KerkinActie -­‐ No author, no date. Manual Monitoring Tool -­‐ Also about monitoring: two untitled and undated notes about the monitoring protocol and one ppt (Decision tree for existing projects – how to use the Monitoring Tool) -­‐ 1558_Compte rendu 21 25 aout 2009_EN_def 21 and 25 August 2009

Annex C. People interviewed Aart van de Broek – Edukans; Linking pin – ICCO Cooperation Mariecke van der Glas -­‐ Regional Manager Nicaragua Dieneke de Groot -­‐ Coordinator PME unit, FACTS department, ICCO/Kerk in Actie Wim Hart -­‐ Executive Board ICCO Cooperation Helmke Hofman -­‐ Specialist Education, Edukans Henk Jochemsen -­‐Directeur Prisma Kees de Jong -­‐ Directeur Edukans Machteld Ooijens -­‐ Head Policy and Development department, ICCO/ Kerk in Actie Anke Plange-­‐van Well -­‐Specialist Health and HIV, PRISMA Piet Postuma -­‐ Specialist CT&D, ICCO/ Kerk in Actie Pepijn Trapman -­‐ Regional Manager South and Central Asia Elly Urban – Prisma; Linking pin – ICCO Cooperation André Vording -­‐ Coordinator-­‐ Specialist FED, ICCO/Kerk in Actie Gerrit de Vries -­‐ Regional Manager Southern Africa Marinus Verweij -­‐ Chairman of the Executive Board ICCO Cooperation Hettie Walters -­‐ Coordinator Capacity development. Policy and Development department. ICCO Cooperation

Annex 3 – Report of the first phase of the Evaluation Programmatic Approach

28

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach -­‐ Survey about the development and performance of coalitions In the context of the evaluation of the programmatic approach of the ICCO Alliance a survey was developed to gain insight in the stage of development of programmatic cooperation through the formation of coalitions. We followed the broad definition of coalitions given in the Guidance Note for the Programmatic Approach: ‘all formal or informal forms of cooperation, alliances, networks, etc.’, which we indicated as ‘coalitions’. The survey was meant for programme officers of the ICCO Alliance (ICCO/Kerk in Actie, Edukans and Prisma) and representatives of ‘coalitions’. The survey was made up of statements to which the respondent could (fully) agree, to some extent or (fully) disagree, posed in three languages: English, French and Spanish. In Box 1 you’ll find the email which invited POs and representatives of a coalition to fill out the survey as well as the introduction section to the survey.

In this report we present the findings in nine sections following the sections in the survey. At the end (in section X) we attempt to answer the relevant questions of the TOR. Attached to the report is a pdf with the overall answers and remarks/ clarifications of the respondents and a pdf which shows a differentiation between the 5 themes. For the analysis use is made of three cross-­‐tables: status of the coalition at its start (‘based on an existing network, coalition, alliance and the like’, versus ‘formation on instigation of ICCO Alliance’); the theme the coalition focuses on (FED, FNS, Basic Box 1: Email inviting POs/ representatives of a coalition to fill out the survey and introduction section of the survey Email Dear program officer/ representative of a coalition supported by the ICCO Alliance, On request of the ICCO Alliance we are evaluating the Programmatic Approach (ICCO/Kerk in Actie; Prisma; Edukans) to help the Alliance in its policy development. We try to involve as m any people as possible to learn about the implementation of the Approach. One of the tools we use is a survey. Since you are key actors in the application of the Approach we very much appreciate to hear your views. Therefore, we kindly invite you to fill out the online survey. We estimate that it takes not more than 20 minutes to complete the questionnaire.

Please click on the following link … to access the survey and SUBMIT THE QUESTIONNAIRE BEFORE ..

AUGUST. We understand that we ask quiet some input from your side and, therefore, we thank you very m uch for your cooperation!

The evaluation team, Verona Groverman Kees Zevenbergen

Introduction section of the survey Background to the survey is survey is part of the evaluation of the programmatic approach of the ICCO Alliance. Presently, the ICCO Alliance does not yet have a good overview of the stage of development of the coalitions it supports. By ‘coalition’ we mean formal or informal forms of cooperation, alliances, networks, etc.

The survey is meant for program officers and representatives of coalitions. It is composed of 9 sets of mostly short multiple choice questions. Please note that a higher score does not automatically mean ‘better’ or ‘more advanced’!

We estimate that it will not take you more than 20 minutes to complete the questionnaire. Please submit the questionnaire by clicking the Send/Submit button at the end BEFORE 15 AUGUST. We will respect confidentiality. The results of the questionnaire will be anonymised so that no individual or organisation will be identified.

Thank you very much for your time and energy, we much appreciate your contribution to this survey!

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

29

health and HIV AIDS, Conflict transformation and Democratization, Basic Education10); and, lastly, member of the ICCO Alliance (ICCO/ Kerk in Aktie, Prisma and Edukans). Where relevant, interesting or striking differences taken from these cross-­‐tables have been added to the findings. However, it appeared difficult to draw hard conclusions about the themes because both the total number of ‘coalitions’ per theme and the number of scores given for certain statements per theme were generally too small.

Section I: General information The survey was sent out to 59 POs and 103 representatives of ‘coalitions’, 162 persons in total11, on 6 August and could be filled out up to 24 August (a reminder was sent to those who had not reacted yet on 16 August). These 162 persons are involved in 95 coalitions totally.

Overall 41 persons submitted the survey, 121 did not react12. The response was as follows13: • 18 representatives of coalitions responded, involved in 17 coalitions.

• 13 POs filled out the survey concerning 12 coalitions.

• In five (5) cases both a representative and a PO filled out the survey for the same coalition.

Concluding, the responses of the survey covered 34 coalitions. The names of these coalitions can be found in annex as well as those coalitions about which the survey did not receive information. In terms of received questionnaires the response rate is 25,3% (30,5% of the POs; 23,3% of the representatives submitted the survey). More important though, is the response rate in terms of coalitions involved since we are specifically interested in the development and performance of coalitions: 35,8%. The picture that can be sketched based on the outcomes of the survey can be considered reasonably representative of the coalitions supported by the ICCO Alliance.

Main features of responses: 1. ‘Coalitions’ are a mix of organisations (respondents could tick more than one option). Over half of the actors involved in the ‘coalitions’ concerned are partners of the ICCO Alliance as shown in figure 1, followed by other NGOs, not being partners of the ICCO Alliance (20%) and government institutions (12%). Table 1 shows the responses per category of actors.

Table 1 -­‐ Actors involved in the ‘coalition’ reactions (%)

Partners of members of the ICCO Alliance (IA 92,7% Other NGOs, not being partners of the ICCO Alliance 36,6% Private sector/business organisations 14,6% Government institutions 22,0% Other (such as knowledge centres, etc.

17,1% question answered

reactions (#) 38 15 6 9 7 41

10

We did not included the theme Fair Climate in the cross table and the analysis of the themes because only one respondent dealt with this theme.

11 The figure is excluding the email address that bounced.

12 One respondent answered 9 of the 15 questions only and it not included in the 41.

13 These figures are not based on the responses in the questionnaire (question 1) but on the list of respondents provided by Survey Monkey. Taking this list made it possible to trace the coalitions in which the representatives and POs were involved and to find out whether or not a PO and representative had assessed the same coalition.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

30

Figure 1 - Actors involved in the coalition

Other (such as knowledge centres, etc) 9% Government ins•tu•ons 12% Private sector/ business organisa•ons 8% Other NGOs, not being partners of the ICCO Alliance/ 20%

Partners of members of the ICCO Alliance (IA) 51%

2. The 34 ‘coalitions’ (some of which regional ‘coalitions’) concerned are mostly located in Africa (19). Others are active in Asia (9), Central Asia (2), South America (3) and in the Middle East (1) (See Annex).

3. Differentiated according to member of the ICCO Alliance we see the following picture – figure 2.

‘Coalitions’ supported by ICCO/Kerk in Actie include private sector organizations and more government institutions and other NGOs than the other Alliance members. In table 2 the actors of the ‘coalitions’ concerned per theme are given.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

31

Figure 2 -­‐ Actors involved in the coalijon per member of the IA

Asjtel

45 40 35 30 25 20 15 10 5 0

Partners of Other Private Governme members NGOs, not sector/ nt of the being business ins•tu•on ICCO partners of organisa•o s Alliance the IA ns

Edukans

4

4

0

0

1

Prisma

12

3

0

2

1

ICCO/Kerk in Ac•e

23

10

6

7

6

Other (such as knowledge centres, etc)

Table 2 -­‐ Actors involved in the ‘coalition’ Fair Food and Basic Conflict Basic Economic Nutrition Health Transform Educatio reactions reactions Develop-­‐ Security and ation and n (%) (#) ment HIV Democrati AIDS -­‐zation

Partners of members of 8 6 13 4 4 92,1% 35 the ICCO Alliance (IA) Other NGOs, not being 4 0 3 3 4 36,8% 14 partners of the IA Private sector/business 4 0 1 0 0 13,2% 5 organisations Government 4 0 3 1 0 21,1% 8 institutions Other (such as 4 0 0 1 1 15,8% 6 knowledge centres, etc)

4. The ‘coalitions’ concerned are mainly supported by the following members of the ICCO Alliance: ICCO/ Kerk in Actie (60% or 24 of the responses), while 30% (12 responses) is supported by Prisma and 15% (6 responses) by Edukans.

5. The majority of the ‘coalitions’ are formed on instigation of (members of) the ICCO Alliance (75,6% of the responses) – see table 3. This holds for all the members of the IA, only four of the 24 ‘coalitions’ supported by ICCO/Kerk in Actie are said to be formed based on existing coalitions or networks, three of which focusing on FED and one on conflict transformation.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

32

Table 3 -­‐ Origin of the ‘coalition’ reactions (%) reactions (#)

the ‘coalition’ is based on a coalition, network or alliance that already existed before the creation of the ‘coalition’ supported 12,2% 5 by members of the ICCO Alliance the ‘coalition’ has been organized on instigation of the members 75,6% 31 of the ICCO Alliance Other 12,2% 5

6. Basic health is the theme on which most ‘coalitions’ concerned focus as shown in Figure 3, followed by Fair Economic development and, in equal percentage, Basic education and Food and nutrition security. The ‘coalitions’ in this survey supported by Edukans (evidently) focus on Basic education, those of Prisma on Basic Health and HIV AIDS while those supported by ICCO/Kerk in Actie focus on a mix of themes (Table 4).

Other 5%

Figure 3 -­‐ Theme of the ‘coalijon’s’ acjvijes

Conflict Transforma•on and Democra•za•on 10%

Basic Educa•on 15%

Fair Economic Development 19%

Basic Health And HIV AIDS 34%

Fair Climate 2%

Food and Nutri•on Security 15%

Table 4 -­‐ Theme of your ‘coalition’s’ activities per member of the IA ICCO/Kerk in Actie Prisma Edukans

Fair Economic Development 7 0 0 Fair Climate 1 0 0 Food and Nutrition Security 6 0 0 Basic Health and HIV AIDS 4 10 0 Conflict Transformation and 4 0 0 Democratization Basic Education 0 2 6 Other 2 0 0

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

33

Section II: Participant diversity and cooperation with other actors Table 5 below shows the responses to six statements concerning the diversity of the participants of the ‘coalition’ and cooperation with other actors. What can we observe?

1. First, how were the potential actors of the ‘coalition’ identified? According to most respondents the initial actors of the ‘coalition’ carried out a stakeholder analysis to identify potential co-­‐ operating parties (statement a: 39,0% score fully agree + 24,3% agree = 63,3%). A slightly lower percentage (58,5 %) states that this analysis was used to form the ’coalition’ (statement b: 26,8% fully agree + 31,7% agree).

2. The growth of the ’coalition’ shows a mixed picture: some have grown organically, others not (statement c). The survey did not allow to dig deeper in this issue. Above (table 3) we have seen that most ‘coalitions’ have been formed on instigation of members of the ICCO Alliance but not how it developed further nor how existing networks developed further. In Box 2 some brief examples of respondents are given on how ‘coalitions’ developed.

Box 2 -­‐ Additional explanations of some respondents on how the ‘coalition’ developed -­‐

-­‐

-­‐

-­‐

The way in which the coalition was formed was driven very much from the perspective of 'this is a requirement in order to receive funding'. At this stage of formation, there was no understanding of what the programmatic approach was nor use of Theory of Change or other MSPs. Also the funded partners are spread across very big geographic areas and in different provinces so collaboration on delivery was hardly possible. So we did the best we could and formed an alliance around learning which has actually worked rather well and we are set to continue post 2015 and are looking at expanding membership and funding. A number of partners start working together during the MFS 1 period, they formed X. X grow and was reorganized (open network organization) into Y. The coalition in X started with 9 partner organisation, over a period of two years the number had dropped to 5 partners. however, the coalition has now started linking up with other networks of donor organisations and also mobilize the organisation that had dropped. The reason for earlier dropping includes end in funding from ICCO alliance for the particular organisation. The coalition partner organisation have a well-­‐established relationship with the local governments in their districts of operation.

En effet, le réseau du programme multi acteur d'éducation non formelle est mis sous fond baptismaux depuis l'année 2007. Tout au début les organisations travaillaient avec le même partenaire mais de manière isolé et chacun son domaine d'intervention et des thématiques similaires. Sentant la nécessité de se regrouper, les organisations de la société civile sous l'impulsion de ICCO ont trouvé opportun de créer le programme multiacteur dont l'objectif est de lutter contre la pauvreté en s'appuyant sur les leviers de l'éducation non formelle, la promotion de filières porteuses, la chaîne des valeurs et la promotion de la santé de la reproduction.

With help of ICCO, the Coalition was initiated by its partners, who have been working on the theme of Basic Health and HIV/AIDS. Later, the linkage/networking has been expanded with other like-­‐ minded network together movement to achieve shared goal.

3. According to all respondents their ’coalition’ is composed of multi-­‐type actors and multi-­‐level actors. This is confirmed by what was shown earlier in figure 1. The respondents have provided all names or abbreviations of the actors involved (not included in this report).

4. Concerning cooperation between activities of the ’coalition’ and other existing networks working on the same problematic, most respondents to some extent or fully agree that this happens (statement f, score 3, 4, 5).

Statements

Table 5 -­‐ Participant diversity and cooperation with other actors 1 3 5 not 4 reactions (fully 2 (to (Fully relevant/ (agree) (#) disagree) some agree) do not

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

34

extent) a. The initial actors of the ‘coalition’ carried out a stakeholder analysis to identify potential co-­‐operating parties b. The ’coalition’ has been formed on the basis of a stakeholder analysis c. The ’coalition’ has grown organically, starting with about two partners and over time increasing the number and type of actors d. The ’coalition’ is composed of multi-­‐type actors e. The ’coalition’ is composed of multi-­‐level actors f. The ’coalition’ carries out activities in close cooperation with other existing networks, working on the same problematic

know

3

4

7

10

16

1

41

4

5

8

13

11

0

41

8

10

6

8

9

0

41

5

2

5

9

18

2

41

4

4

9

9

13

2

41

2

6

10

12

10

1

41

Section III: Implementation and implementation capacity of the ‘coalition’ Through a few statements we tried to get an impression of how the ‘coalitions’ plan and implement their activities. Table 6 shows the scores given by the respondents on various statements, which were mostly positively assessed (score 4). Our main observations are the following.

1. It appears that most members of the ’coalitions’ organise their planning based on a logframe (statement a). In three cases such a planning does not exist (score 1). Less ’coalitions’ have an organised planning based on a Theory of Change (statement b). There could be different explanations of these outcomes. For example, the ‘coalitions’ now use a Theory of Change (after the trainings conducted by the Alliance) while previously using a logframe. Or, they do both following requirements of different donors. One respondent explained: Our work as a coalition is very shaped and bounded by the MFS II Results Framework. However, we are now starting to think about and plan work outside of this framework.

2. The majority of the respondents (65,8%) agree that the ’coalition’ reaches its targets and objectives (score 4, statement c). Less (36,6%) indicate that the ‘coalition’ has a monitoring system in place that is used, while 29,2% does have not a proper system in place/does not use it.

Looking at the start situation of the ‘coalition’ less ‘coalitions’ that are based on existing networks, coalitions etc. indicate to have a monitoring system in place that is used than those formed on instigation of the IA , but the total numbers are too small to draw hard conclusions (it concerns 5 ‘coalitions’ in total).

3. Most of the respondents agreed on the statement staff of all actors in the ’coalition’ has sufficient expertise to address the issues on which the cooperation focuses (statement e). It may be well in line with their perceptions on the previous statements mentioned under point 2: planning is well organised and targets / objectives are reached.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

35

4. The statements on how activities are being implemented appeared confusing: both the statement (f) Activities planned by the ’coalition’ are carried out by individual actors and (g) Most activities planned by the ’coalition’ are carried out jointly got the highest response in score 4. Most respondents (58,5%) were positive about the flexibility in which the ‘coalition’ adapts its activities based on learning (statement h -­‐ score 5, 3 and 4 in decreasing response), but still 26,8% scored to some extent: 3. In Box 3 a few additions by respondents are given.

Box 3 -­‐ Additional explanations of two respondents on implementation issues -­‐

-­‐

La coalition est structurée de manière qu'il est mis en place d'un comité de pilotage et d'une commission technique. Pour une meilleure coordination des activités de la coalition, le comité de pilotage a mis en place une équipe chargée de mettre en oeuvre la mission déclinée. il s'agit de la coordination qui a en charge de développer des activités de plaidoyer lobbying, de renforcement des capacités des acteurs et de suivi évaluation des activités. la coordination rend compte au comité de pilotage périodiquement de ses activités qui fera l'objet de validation.

The coalition implements commonly agreed activities using the existing technical expertise among the member organisations. It is also common for partner organisation to share expertise e.g. in training, follow up etc. Although the individual organisations have their monitoring systems, a joint monitoring mechanism has not been fully developed (indicators were developed).

5. About half of the respondents state that mechanisms for accountability towards target-­‐groups, clients or grassroots representatives are in place, but still 26,8% scored to some extent (3). One respondent added that one partner uses the Client Satisfaction Instrument.

Table 6 -­‐ Implementation and implementation capacity of the ‘coalition’ not 1 5 4 relevant; (fully 2 3 (Fully

(agree) do not disagree) agree) know/ a. The ’coalition’ has a well organised planning based on a 3 6 9 8 14 1 logframe b. The ’coalition’ has a well organised planning based on a 1 5 13 15 7 0 Theory of Change c. The ’coalition’ reaches its 0 3 5 27 6 0 targets and objectives d. The ‘coalition’ has a monitoring 2 10 9 15 5 0 system in place that is used e. Staff of all actors in the ’coalition’ has sufficient expertise 1 3 8 18 11 0 to address the issues on which the cooperation focuses f. Activities planned by the ’coalition’ are carried out by 6 3 9 15 7 1 individual actors g. Most activities planned by the 2 8 7 13 11 0 ’coalition’ are carried out jointly h. The ‘coalition’ flexibly adapts its set of activities based on learning 1 5 11 10 14 0 on what works and does not work i. Mechanisms for accountability towards target-­‐groups, clients or 0 8 11 12 10 0 grassroots representatives are in Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

reactions (#)

41

41 41 41

41

41 41 41

41

36

place

Section IV: Attention to gender equality The survey included four statements to identify the extent to which the ‘coalitions’ are paying attention to gender equality, a cross-­‐cutting theme of the ICCO Alliance reflected in a policy. 14 Table 7 gives the results. We like to point to the following results.

1. Concerning a clear vision on gender equality of the ‘coalition’ the respondents’ scores mostly vary between 3, 4 and 5: 26,8%, 31,7%, and 31,7% respectively (statement a). We see the same variation on the ‘implementation’ of such a vision – efforts to decrease gender inequality -­‐ with only slightly different responses: 24,3%, 31,7% and 34,1% (statement b). Monitoring the impact of its activities on gender equality is not done by all ‘coalitions’: most (41,4%) score ‘to some extent we do so’ (score 3), and 36,5% scores 4 + 5: we (fully) agree that we monitor. In Box 4 five reactions of respondents are added which shows the variation between the ‘coalitions’.

Box 4 – Additions by five respondents on gender equality -­‐ -­‐ -­‐ -­‐ -­‐

-­‐

Cooptation de X spécialiste du gender in value chain The coalition has not yet reflected on the gender dimension as common area of concern. Individual partner organisations have specific actions that are geared towards addressing the gender inequality La vision stratégique de la coalition repose essentiellement sur l'équité et l'égalité de genre en termes de contribution à la stratégie nationale pour l'équité et l'égalité de genre au X. C'est pour cette raison que le but est de lutter contre les inégalités sociales et la pauvreté des jeunes et des femmes.

One of the weak points, but at present a gender specialist is involved as well as a gender specific organization as a member of the platform.

Gender is a cross cutting issue for the coalition and each coalition member has a gender based policy to establish a gender equality system in the respective organization.

2. The number of ‘coalitions’ that include actors with a track record in dealing with gender inequality is almost the same for the score 2 – 5, about 23% each, in other words, there is much difference between the ‘coalitions’.

14

Unlearning gender. ICCO Alliance gender policy. 2010

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

37

Table 7 -­‐ The extent to which the vision, objectives and strategy/ activities of the ‘coalition’ include attention to inequality between the two genders not 1 5 4 relevant; reactions (fully 2 3 (Fully

(agree) do not (#) disagree) agree) know a. The ’coalition’ has a clear vision on equality between 1 3 11 13 13 0 41 men and women b. The activities of the ‘coalition’ include efforts to 0 4 10 13 14 0 41 decrease inequality between men and women c. The ’coalition’ monitors the impact of its activities on 1 7 17 9 6 1 41 inequality between men and women d. The ’coalition’ includes actors with a track record in 3 8 10 11 9 0 41 dealing with gender inequality

Section V: Use of a rights-­‐based perspective Similar issues as for gender equality were raised concerning the use of a rights-­‐based perspective.

ICCO adheres to the APRODEV formulated RBA policy.15 Attention to RBA is also paid in the Alliance’s gender policy. The scoring on the issues can be found in table 8. Some main observations: 1. Concerning a clear vision the respondents score a bit lower than on gender equality: the majority scored 3 and 4: 34,1% and 31,7% respectively. According to the respondents most activities of the ‘coalition’ include efforts to strengthen the capacities of rights holders, especially vulnerable groups: 78% for score 4 and 5 combined.

2. A number of respondents 13 (31,7%) fully agree that activities of the ‘coalition’ include efforts to hold duty bearers accountable, which makes 56% when score 4 and 5 are combined. But the number that score ‘we do this to some extent ‘(score 3) is still high: 15 (36,6%).

3. The highest score concerning the ’coalition’s’ monitoring of the impact of its activities on vulnerable groups is 4, namely 51%.

15

http://www.aprodev.eu/files/Development_policy/Dev-RBA/Rights-Position-Paper_E-2008.pdf

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

38

Table 8 -­‐ To what extent does the ’coalition’ use a rights based perspective

a. The ’coalition’ has a clear vision on the obligations of duty bearers and the entitlements of rights holders, especially of vulnerable groups b. The activities of the ‘coalition’ include efforts to strengthen the capacities of rights holders, especially vulnerable groups c. The activities of the ‘coalition’ include efforts to hold duty bearers accountable d. The ’coalition’ monitors the impact of its activities on vulnerable groups

3

4 (agree)

5 (Fully agree)

not relevant; do not know

reactions (#)

2

14

13

10

0

41

1

0

8

14

18

0

41

1

2

15

10

13

0

41

1

1

11

21

6

1

41

1 (fully disagree)

2

2

Section VI: Ownership and complementarity of the ’coalition’ In the survey statements have been included to better understand the actors’ commitment to the ‘coalition’. Issues such as decision making, feeling of ownership and responsibility, the relation between the own organization and the ‘coalition’ are helpful indicators in this respect. An overview of the scores is given in table 9.

1. The highest scores for all the statements are found under score 4 and/or 5. Taking score 4 and 5 together 51% of the respondents say that decisions within the ’coalition’ are taken without prior consulting of the ICCO Alliance and even more that a feeling of responsibility exists for the cooperation (68%). In Box 5 some additions by respondents can be read. 65,8% of the respondents state that actors of the ’coalition’ feel ownership of their cooperation and its results (combined scoring on 4 and 5).

Box 5 – Additions by respondents on decision making -­‐ -­‐

-­‐

Cependant, il faut préciser que certaines décisions concernant les éléments contractuels nécessitent la consultation préalable de l'Alliance ICCO car c'est contractuel.

Il est vrai que l'alliance X est créée mais elle dépend entièrement de ICCO encore pour développer ses actions sur le terrains. ICCO est partenaire de l'alliance mais elle a encore besoin du soutien de ICCO aussi bien sur le plan technique que financier.

Our decision on the annual plans is based on the bi-­‐annual reflection meetings where partner present their wish-­‐list/challenges and area of common interventions are identified. We have the decision making team in place to take day to day decisions on behalf of the coalition.

2. To many respondents (73% -­‐ scoring 4 + 5) the ’coalition’ is complementary to what individual actors are doingThe scoring on identification of clear win-­‐win effects of the cooperation shows a mixed picture: from score 3 to score 5 (26,8%, 26,8%, and 31,7% respectively). One

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

39

representative further explained: The win-­‐win effects refer to results of actions whereby stakeholders feel satisfied at some extent and feel fully part of them.

Table 9 -­‐ Ownership and complementarity of the ’coalition’ 3 5 1 4 not relevant; reactions 2 (to some (Fully

(fully disagree) (agree) do not know (#) extent) agree) a. Decisions within the ’coalition’ are taken without prior 5 4 10 13 8 1 41 consulting of the ICCO Alliance b. Actors of the ’coalition’ feel ownership of their 1 5 7 14 13 1 41 cooperation and its results c. Actors of the ’coalition’ feel 1 2 10 12 16 0 41 responsible for the cooperation d. The ’coalition’ is complementary to 1 1 8 12 18 1 41 what individual actors are doing e. There are clear win-­‐win effects of 0 6 11 11 13 0 41 the cooperation identified by all

3. As mentioned earlier, it is difficult to assess differences between and within themes due to the small numbers. Interesting though is that for statement a) Decisions within the ’coalition’ are taken without prior consulting of the ICCO Alliance both the themes Food and Nutrition security and the Basic education score lower than the other themes.

Section VII: Financial sustainability Respondents were asked to provide information about financial issues related to the ‘coalition’ in order to gain insight into their financial sustainability. The responses can be found in Table 10 below.

Our main observations are the following.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

40

Table 10 -­‐ To what extent are the ’coalition’ and its participants financially sustainable not 1 5 4 relevant; reactions (fully 2 3 (Fully

(agree) do not (#) disagree) agree) know a. The actors in the ’coalition’ rely 1 7 13 9 11 0 41 on funds of the ICCO Alliance b. Financing of activities of the ’coalition’ is largely carried out 16 12 9 1 2 1 41 with own sources c. A financial strategy of the ‘coalition’ has been developed 6 15 13 3 3 1 41 based on diversification of resources d. Financing of activities of the ’coalition’ is largely carried out 9 15 9 4 3 1 41 with diverse sources

1. Activities of the ’coalition’ are hardly carried out with own sources (statement b). This holds for all of the themes. It matches with the responses to statement d: only 17 % (fully) state that activities of the ‘coalition’ are largely carried out with diverse sources. Moreover, 48,8% of the respondents state that the actors rely on funds of the ICCO Alliance (statement a, score 4 + 5 combined). As far as the small numbers of ‘coalitions’ per theme can tell, it seems that especially Basic health and Basic education rely on ICCO Alliance funds.

2. Almost none ‘coalition’ has developed a financial strategy based on diversification of resources (Score 2 and 3: 36,5% and 31,7% respectively). Only a few ‘coalitions’ focusing on FNS (2 of 6 ‘coalitions’), Conflict transformation and democratization (3 of 4), and Basic education (1 of 6) do so. Financial management is an area that is not developed fully and needs more attention according to remarks added by a few respondents – see Box 6 below.

3. Comparing the responses of ‘coalitions building on existing coalitions, networks and the like’, with those of ‘coalitions formed on instigation of the ICCO Alliance’, it appears that the first type scores a bit higher than the latter. It seems that to some extent (score 3) they carry out activities with their own resources and with diverse resources and have developed a sort of financial strategy towards diversification of resources.

Box 6 – Additional remarks about financial management -­‐ -­‐ -­‐ -­‐

-­‐ -­‐ -­‐

-­‐

This is still a developing but growing area that is still very slow We need to develop this areas in future La coalition est dans un processus de restructuration. la coalition mettra en place dans les prochains jours un plan de mobilisation de ressources.

The partners in coalition were trained on financial Management including financial strategies, fundraising, procurement procedures, etc. One of the requirements in each Contract is fundraising and annual reduction of funds from ICCO/DCA Presque toutes les activités de la coalition sont financées par l'Alliance ICCO. Cependant, la coalition a élaboré un plan pour assurer le funraising et la mobilisation des fonds.

They rely for the Education Program mainly on funding from ICCO. Almost all partner have other donors for their programs / activities.

L'alliance sécurité alimentaire dépend entièrement de ICCO encore pour le financement de ses activité.

Un défi important sur lequel nous travaillons est la diversification des sources de financement.

We have started discussions about diversifying our funding. We have approached two donors and plan to continue developing a diverse funding base for the coalition.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

41

Section VIII: The internal functioning of the 'coalition' About nine statements were included in the survey to find out to what extent internal conditions for joint working and action are in place. The results are reflected in table 11 below. Some main observations: 1. A number of internal conditions for joint working and action are in place: -­‐ Actors’ competences (statement a): respondents mostly agree that the actors of the ’coalition’ have most competencies (networking, conflict resolution, interpersonal skills) to function smoothly (48,7% -­‐ score 4); -­‐ Trustful relationships (statement b): according to most respondents relationships between the actors of the ’coalition’ are built on trust (score 4 + 5 together 78% of the responses); -­‐ Definition of roles (statement d): the roles between the actors of the ’coalition’ are well-­‐ defined, building on each other’s strengths (score 4 + 5 together 68% of the responses); A respondent added: Avec le manuel des procédures, le fonctionnement de la coalition est organisé par des instruments qui définissent les rôles et responsabilités de chaque membre.

-­‐ Communication strategy (statement c): 63,4% of the respondents state (score 4+5 together) that the ’coalition’ has put an effective internal communication strategy between participating actors in place, but still 29,3% are indecisive (score 3); -­‐ Participation (statement e): 48,7% of the respondents agree (score 4) that the actors of the ’coalition’ participate actively to ensure smooth functioning of the cooperation; -­‐ Governing body (statement g): in 46,3% (score 5) of the ‘coalitions’ a representative governing body is said to be in place. A respondent added: Internal structure is in place and functions well: Coordination Council, thematic sub-­‐groups: internal migration, external migration, sending communities, focal points by regions, Information Committee, General Meeting. Comparing the responses of ‘coalitions building on existing coalitions, networks and the like’ with those of ‘coalitions formed on instigation of the ICCO Alliance’, a considerable number of respondents of the latter ticked score 3 (to some extent) as well (10 out of 31 responses, score 4: 13 out of 31) -­‐ Internal learning (statement h): respondents (fully) agree that the actors of the ’coalition’ make active use of learning opportunities within the ’coalition’ (score 4 +5 together 68,3%).

Two respondents clarified: The coalitions conduct peer to peer visits and also share technical expertise in trainings; and, Within the coalition, the partners had exchange programmes within the partnership to learn from each other the best practices and implement them in their respective programme areas.

2. Still a number of respondents ticked score 3 (to some extent) as shown in the table. In particular, the scoring on formalisation/ implementation of a joint governance or co-­‐responsibility of the ’coalition’ are almost equal for score 3, 4, and 5 (around 29%) (statement f). The same equal scoring applies to actors’ active use of learning and exchange opportunities outside the ’coalition’.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

42

Table 11 -­‐ The internal functioning of the 'coalition': To what extent are internal conditions for joint working and action in place 1 5 not 4 reactions (fully 2 3 (Fully relevant; do

(agree) (#) disagree) agree) not know a. The actors of the ’coalition’ have most competencies (networking, conflict resolution, 1 3 9 20 8 0 41 interpersonal skills) to function smoothly b. Relationships between the actors of the ’coalition’ are built 1 2 6 16 16 0 41 on trust c. The ’coalition’ has put an effective internal communication 0 3 12 18 8 0 41 strategy between participating actors in place d. The roles between the actors of the ’coalition’ are well-­‐ 0 5 7 13 15 1 41 defined, building on each other’s strengths e. The actors of the ’coalition’ participate actively to ensure 2 3 8 20 8 0 41 smooth functioning of the cooperation f. Joint governance or co-­‐ responsibility of the ’coalition’ 2 2 11 12 13 1 41 is formalised and implemented g. A representative governing 3 3 6 9 19 1 41 body is in place h. The actors of the ’coalition’ make active use of learning 0 4 9 14 14 0 41 opportunities within the ’coalition’ i. The actors of the ’coalition’ make active use of learning and 0 6 11 11 13 0 41 exchange opportunities outside the ’coalition’

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

43

Section IX: Effects of ’coalition’ or cooperation A last series of questions were posed to gain insight into the perceived effects of the ‘coalition’ or the cooperation between its actors. For the outcomes see table 12 below. We did not observe differences between coalitions that already existing before they received from support from the ICCO Alliance and those there were formed on instigation of the Alliance. Respondents were asked to add an example of an effect– see some responses in Box 7. Some main observations are given below.

1. A number of pre-­‐defined effects were given which the respondents could score: -­‐ Organizational development of the individual actor (statement a): almost the same number

-­‐

-­‐

-­‐

-­‐

-­‐

of respondents indicated that such development took place as those who said it happened ‘to some extent’– about 31%; the other scores were lower.

Increased realization among actors that jointly undertaking activities has added value (statement b): the majority of the respondents ticked agree or fully agree (score 4+5 together 75%).

Increased realization among actors that joint fundraising has added value: again, almost the same number of respondents agreed to this statement c as those who felt such happened ‘to some extent’ – about 26%; We have seen in section VII that most ‘coalitions’ do not yet have a financial strategy.

Increased realization among actors that joint learning has added value (statement d): the highest scoring concerns score 4 and 5 ( both scores together 75%) This is in line with the views on internal learning indicated in section VIII: the actors of the ’coalition’ make active use of learning opportunities within the ’coalition’.

Purposely member diversification to effectively address the root causes of poverty and injustice (statement e): here, most respondents agree (32,5%) or score that they do so to some extent (40%).

The overall scoring concerning ‘Sustainable forms of cooperation between different actors are increasingly developing due to the programmatic approach’ (statement f) shows that most respondents agree with this statement (42,5%) with almost equal number scorings on 3 and 5 (25% and 22,5 % respectively).

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

44

Table 12 -­‐ Effects of ’coalition’ or cooperation 1 5 not 4 reactions (fully 2 3 (Fully relevant; do

(agree) (#) disagree) agree) not know a. Organisational development takes place by individual actors 1 5 12 13 9 0 40 due to the participation in the ’coalition’ b. The actors of the ’coalition’ increasingly realise added value 1 2 7 16 14 0 40 in jointly undertaking activities c. The actors of the ’coalition’ increasingly realise added value 4 8 10 11 7 0 40 in joint fundraising d. The actors of the ’coalition’ increasingly realise added value 1 0 9 16 14 0 40 in terms of joint learning e. The ‘coalition’ varies its member composition to 5 8 10 13 3 1 40 effectively address the root causes of poverty and injustice f. Sustainable forms of cooperation between different actors are increasingly 1 3 10 17 9 0 40 developing due to the programmatic approach

Box 7 – Examples of effects of cooperating in a ‘coalition’ Capacity building/Exchange -­‐ Cooperation on training, advocacy and lobby.

-­‐ Capacity building that include accompaniment by skilled staff and experience exchange -­‐ Within the coalition the 'district approach' of one of the partners, with involvement of district organisations in training and capacity building, is taken over by other partners.

-­‐ La realización conjunta de un proceso de sistematización de las experiencias de formación de docentes en servicio, con enfoque de Educación Intercultural Bilingüe.

Joint fundraising -­‐ Some of the partners within the network have jointly written successful proposals, eg …. Other partners have shared office space and staff with fellow network partners, eg .. Other partners have strengthened their cost-­‐ recovery mechanisms, eg ….

-­‐ Soumission conjointe d'une proposition de projet auprès de l'Union Européenne (X): les membres de la coalition ont formulé d'une manière participative et conjointe le projet. L'intervention aur lieu dans un site commun, les apports-­‐ contribution propre exigé par l'appel à proposition seront supportés par les membres.

-­‐ The coalition has opened up to multiple funders.

Cooperation in activities -­‐ Cooperation between .. Regional producers organisations & … a national Represent Process in factory and ..

National NGOs and … international development partners.

-­‐ Actualmente, X se encuentra en una etapa de Fortalecimiento, después de 3 años de su conformación. Se están construyendo poco a poco los mecanismos de cooperación, interacción, compromiso, negociación y monitoreo dentro del emprendimiento, considerando el compromiso que todos van asumiendo dentro de esta nueva instancia de articulación. No ha sido fácil, pero se está creando un reconocimiento de parte de todos los socios al ir buscando nuevos proyectos y oportunidades de manera a ir cubriendo las necesidades identificadas dentro de la organización, en el sector orgánico y de cada uno de los integrantes.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

45

-­‐ Partnership building between coalition members, influenced government authority to develop policies to integrated address HIV and health related issues through integrating it into government health and development system.

-­‐ Actors working in the same geographical area increasingly coordinate their interventions with each other.

-­‐ Dans le traitement des problématiques liées à la SAN, Y se mobilise (i) en menant des actions de plaidoyer des différents niveaux (régional, national et international) en (ii) en développant des projets visant à renforcer et améliorer la SAN des couches vunlérables. chaque entité membre remonte au niveau du Secrétariat Exécutif les questions nécessitant une intervention conjointe pour avoir plus des voix et pour être écoutés.

Diversification of the ‘coalition’ / networking -­‐ ICCO est en partenariat durable avec …. Toutefois, la Coalition .. commence en 2011 et est un debut de cooperation durable parce que le partenariat est diversifie et complementaire avec ces 8 membres de la Coalition.

-­‐ L'exemple que nous pouvons citer pour assurer une durabilité de la coopération constitue la mise en réseau des partenaires techniques et financiers.

-­‐ Establishing thematic platforms/networking (established many addiitonal specific thematic platforms with government, institualization of various cooperation). For example, in X the platform has established a ..Public Council on migration including parliamentarians, ministries, NGOs -­‐ within the thematic of sending communities. or Established "Regional Forum" (multi stakeholder, multilevel) within the group of external migration.

-­‐ Besides working with the ICCO Cooperation, the program now seeks linkages with other partners in the ACT Alliance (e.g. Christian Aid).

Outcomes of activities -­‐ The Community Groups at the village level are getting more aware about their roles and responsibilities and collective action to address their needs such as relevant and quality education. They are also able to demand services from the government.

Other -­‐ For example, X provides recorded programmes on health for community listener groups at Y. -­‐ Par le biais du programme des ONG pour la sécurité alimentaire et nutritionnelle au A, les ONG membres de l'alliance Sécurité alimentaire ont obtenu plus de visibilité dans leurs actions.

-­‐ Les membres de la coalition ont décidé de restructurer le reseau pour devenir une organisation stable et durable.

-­‐ Where roles of all the actors and coalition vision are clearly defined and decisions are participatory made to promote ownership of the coalition among the actors.

Lastly, a few respondents added some remarks concerning cooperation and coalition building and the survey exercise itself – these are included in box 8.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

46

Box 8 -­‐ Additional remarks on the survey/issues addressed in the survey • Promoting coalition is the way to address root causes of poverty and injustice. However, ensuring its effectiveness takes time and requires resources. There is high level of commitment from ICCO Cooperation towards coalition building. The partners (local actors) also started getting better understanding on coalition building. However, still there is long ways to go. We started with our (ICCO Cooperation's partners) and the diversity among the members of coalitions that we initiated is minimal. If I take the case of the coalition I am working with it consists eight partners all funded by ICCO Cooperation and so far they couldn't bring on board non-­‐funded actors. On the other hand for the coalition to bring intended changes, there is a need to have different types of actors at different levels. Hence, this should be an area where we need to focus on for the future. I work with two coalitions, but since generally the two are similar, filling the survey twice will not add any value.

• It is a relevant exercise and should be done periodically.

• Ce genre de suivi permettra de procéder au renforcement organisationnel de la Coalition • Thank you for the opportunity of assessing/reflecting ourselves.

• Nous avons prevu un atelier regional entre les partenaires de .. du … pour partager les experiences entre les 3 pays. Aussi, nous prevoyons la Theorie du Changement pour la Coalition …..

• This survey is very important to jauge the progress of coalitions towards the programmatic approach. It's also an eye opener regarding things that have been achieved compared to what should be done.

• C'est un questionnaire assez compréhensible et explicite.

• In addition to common activities within the coalition, there are emerging joint working among 2-­‐3 organisations within the coalition to address a specific issues probably internal within those organisation or within a specific region/area even outside the common programme operation region. So cooperation within the coalition is also possible and should not be seen as a problem. Resource for the coalition to operate for the near future is very important, however, this could be followed with a Fundraising strategy that can also consider financial resources within the country and also from donors outside ICCO alliance. otherwise the immediate results show that there is win-­‐win situation for most partner organisation in the area of learning and sharing.

• X is the lead agency in Y Beyond 2015 to lead, initiate and facilitate the national consultation process/deliberations among civil society organizations to come out with post 2015 MDG development framework.

Section X: Analysis against the relevant statements of the TOR The TOR requested more detailed information on various items about which the survey, to a certain extent, has provided some insight. We address them below.

• Evolution of the programme coalitions and variations thereof in various thematic and geographical settings The survey does not give a picture on how the ‘coalitions’ have developed because it did not include a time dimension. From a geographic point of view, most responses have been provided for African ‘coalitions’. Thematically, most responses were provided for the theme Basic Health and HIV AIDS.

• Roles and responsibilities of the various members of the coalitions and the ways in which these have evolved (including ICCO staff and external advisors) The survey gives some information on roles and responsibilities in terms of clarity about such roles.

Respondents were asked about the extent to which the roles between the actors of the ’coalition’ were well-­‐defined, building on each other’s strengths. In section VIII we have seen that the majority agrees that this is the situation in their ‘coalition’ (score 4+5 together 68,37%). The survey did not ask about the specificities of the roles and responsibilities and those of ICCO staff and external advisors.

• The sustainability of the cooperation within the programme coalitions/ The effectiveness of the programmatic approach as a methodology to strengthen sustainable forms of cooperation between different actors in Southern countries The survey included two statements about sustainability. First, about financial sustainability (section VII). The survey revealed that activities of the ’coalition’ are hardly carried out with own sources. This holds for all of the themes. Almost half of the respondents (48,7% scored 4+5) indicated that the Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

47

actors of the ‘coalition’ relied on funds of the ICCO Alliance, while a much smaller group said that their ‘coalition’ uses diverse resources among which IA funds (17% score 4+5). Almost none ‘coalition’ has developed a financial strategy based on diversification of resources. Financial management is an area that is not developed fully and needs more attention according to remarks added by a few respondents. It is in line with the responses to the statement that there is Increased realization among actors that joint fundraising has added value (about 26% of the respondents agreed to this statement and the same percentage indicated that they did so to some extent (score 3). Comparing the responses of ‘coalitions building on existing coalitions, networks and the like’ with those of ‘coalitions formed on instigation of the ICCO Alliance’, it appears that the first type scored a bit higher than the latter. The survey did not dig deeper in the reasons behind this observation.

Second, the question was asked about sustainable forms of cooperation due to the programmatic approach: Sustainable forms of cooperation between different actors are increasingly developing due to the programmatic approach (Section IX). The answer indirectly tells something about sustainability of the ‘coalitions’: most respondents can agree with this statement (65%) with a much lower percentage in score 3 (25%). This holds for all themes. Other survey statements can be considered proxy indicators of sustainability of ‘coalitions’: o On Effects of ’coalition’ or cooperation (Section IX): Most respondents indicated that there is an increased realization among actors that jointly undertaking activities and joint learning both have added value. o On Implementation and implementation capacity of the ‘coalition’ (Section III): The majority of the respondents agreed that the ’coalition’ reaches its targets and objectives. Since not all ‘coalitions’ have a monitoring system in place that is used it is not clear on which this perception is based. Most respondents were also positive about the flexibility in which the ‘coalition’ adapts its activities based on learning. Furthermore, most respondents perceived the staff of all actors in the ’coalition’ sufficiently capable to address the issues on which the cooperation focuses. o On Ownership and complementarity of the ’coalition’ (Section VI): Over half of the respondents (51,2%) indicated that decisions within the ’coalition’ are taken without prior consulting of the ICCO Alliance. According to the majority of the respondents (about 68%) a ‘feeling of responsibility’ exists for the cooperation as well as a ‘feeling of ownership of the cooperation and its results’. This finding links well with one of the ‘pearls’ discussed in the report of the first phase of the evaluation.

• The ways in which and the extent to which gender equality and human rights are being integrated / taken up in programmes Two sections in the survey addressed the issues: Section IV and V. The overall conclusions are that over 60% of the respondents feel that the ‘coalition’ has a clear vision on gender equality and make efforts to decrease gender inequality, while about 25% have a mixed feeling (score 3). Concerning a clear vision on a rights-­‐based perspective, the respondents score a bit lower than on gender equality but still over 50%. However, according to the respondents most activities of the ‘coalition’ include efforts to strengthen the capacities of rights holders, especially vulnerable groups (78% scored 4+5 together). Also, the majority (56%) states that activities of the ‘coalition’ include efforts to hold duty bearers accountable. Furthermore, not all ‘coalitions’ monitor the impact of its activities on gender equality (36,6% score 4+5) but based on the responses many ’coalitions’ monitor the impact of its activities on vulnerable groups (65,8% score 4+5). This result is interesting because 48,8% of the respondents agree that the coalition has a monitoring system in place and 29,3% even denies that there is such a system (SectionIII).

• The most significant results of working with the programmatic approach to date For all themes almost all respondents gave at least score 3 and mostly higher about the programmatic approach: Sustainable forms of cooperation between different actors are increasingly Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

48

developing due to this approach. The survey also includes some perceptions on the results of the ‘coalitions’ which could be considered proxy indicators on significant results – the effects of ’coalition’ or cooperation which we already addressed above. Also, respondents reacted to the statement There are clear win-­‐win effects of the cooperation identified by all. The scoring on this statement showed a mixed picture ranging from indecisive and fully agree (score 3 – 5). The survey did not ask to clarify the effects or to give examples.

Verona Groverman, Kees Zevenbergen, 9 October 2013

Annex -­‐ List of coalitions about which information was collected through the survey (34) 1. ALOCES E-­‐DR Congo 2. COSPASAK Coordination of the PASAK Program (Program d’Appui a la Securite Alimentaire du Sud Kivu) 3. ICCO Alliance Health program Ethiopia 4. SSHARE South Sudan 5. ICCO Alliance Health program Uganda 6. Lango FNS Cluster Uganda 7. Distaster Risk Reduction Platform for Teso (DRRP4T) Uganda 8. Central Asia on the Move (CT&D) Kyrgyzstan, Tadjikistan, Russia and Kazakhstan 9. Program Gran Chaco (CT/D) Bolivia, Argentina, Paraguay 10. Association Paraguay organic 11. Ruta del Sol Peru 12. Afghanistan Health Alliance 13. South Odisha Development Initiative India 14. Development Focus, Bangalore India 15. Western Orissa Education Watch (WOEW) 16. Health Bridge Alliance India 17. HIV/AIDS & SRHR Coalition Nepal 18. Seaweeds Net (Samar Island Seaweeds Value Chain Network) Philippines 19. PhilSEN (Philippine Social Enterprise Network) 20. Vietnam River Network 21. Land and Forest Coalition Vietnam 22. Coalition LIFE Madagascar 23. Uchembere Network Malawi 24. National Rice Development Platform Malawi 25. SALT Alliance South Africa 26. Basic Health & HIV/Aids Coalition Zimbabwe 27. Coalition Sécurité Alimentaire et de Nutrition du Burkina Faso (C/SAN) 28. Alliance Programme Spécial de Sécurité Alimentaire et Nutritionnelle du Bénin (ProSSAN) 29. Alliance Soja du Bénin 30. Alliance Anacarde du Bénin 31. Ghana WASH Alliance 32. Alliance for Strengthening Education in Ghana (ASEG) 33. Programme Multi Acteurs ENF-­‐SRDS/J Senegal 34. Palestinian Human Rights Organisations Council (PHROC) Defense for Children International Palestine

No information was received about the following coalitions: 1. Amhara Cluster Ethiopia 2. Oromiya Cluster Ethiopia 3. Afar Cluster Ethiopia Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

49

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

33.

34.

35.

36.

37.

38.

39.

40.

41.

42.

43.

44.

45.

46.

47.

48.

49.

50.

16

WASH Uganda, Kenya, Ethiopia TUCO Kenya STAR School Programme Coalition Kenya Food security and thematic program consortium of the European Union Republic of South Sudan Western Bahr el Ghazal state Basic Health and HOV/Aids GZB, Light for the World, Dorcas South Sudan Centroamerica Democratica (CAD) Guatemala, El Salvador and Honduras Programa de Seguridad Juvenil (PSJ: Juvenile Security Program) Mesoamerican Alliance of Peoples and Forests FOROLACFR Latin American Forum for Rural Finance REDIMIF Microfinance Network of Guatemala RED KATALYSIS Central America ( Guatemala, Honduras, Salvador, Nicaragua, Costa Rica) AGEXPORT Guatemalan Export Asociation Association of Microfinance Institutions in Kyrgyzstan Coordination Council on Development of microfinance in Kyrgyzstan RIMISP Latinamerican Center for Rural Development SUSO – VSC Brazil, Bolivia, Peru Gran Chaco Program Regional Food Security Programme focusing on the Human Right to Adequate Food Perú, Bolivia y Paraguay Consortium Agro ecological Peru Bangladesh WASH Alliance Health Alliance Bangladesh Vikas Bazar Network India Siksha se Parivartan (SSP) – Jharkhand India Siksha Chetana (SC) -­‐ Orissa DAHAR – Jharkhand India Bastar Ujar – Chhattisgarh India National Coalition on Food and water security (NCFAW) Nepal Towards Ecological Recovery Regional Alliance (TERRA) Mekong Region (SEA) Asia Indigenous Peoples Pact (AIPP) National Rural Women’s Congress (NRWC) Philippines Mindanao Peace Partners Philippines RightsNet Philippines NOMIA (Negros Island Muscovado Industry Association) Philippines PhilNet-­‐RDI (Philippine Network of Rural Development Institutes) PACCI (Provincial Access Centers Consortium, Inc) Philippines PFEC (Philippine Federation for Environmental Concerns) FuND Philippines (Fundraisers’ Network for Development) OCCP (Organic Certification Center of the Philippines Philippines) Organic rice Northern Mindanao Organic rice Southern Mindanao Microfinance Council of the Philippines Bicol Microfinance Council Philippines Mindanao Microfinance Council National Confederation of Cooperatives Philippines Name? Agriculture ANGOLA ICPM16 Madagascar AINA17 Madagascar

Initiative Commune pour le Plaidoyer à Madagascar : Consortium of four international NGO: CARE, Medair, FAO, MdM, ICCO. Currently in its 3rd phase. ICCO participates in collaboration with its local partner SAF/ FJKM.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

50

51. PAMOLEA18 52. Health coalition Malawi 53. Programme multi acteurs d’éducation, de formation professionnelle et de la promotion de la SRDS/AJ Burkina 54. Alliance Ananas du Bénin 55. Alliance WASH-­‐Bénin (AWB) 56. Program Multi Acteurs d’Alphabetisation Mali 57. FSN coalition Mali 58. Who Profits? Economic research together with partners Israel 59. United Civilians for Peace (UCP) 60. OPGAI 61. Kairos oPt and ww

17

Actions Intégrées en Nutrition et Alimentation. Consortium of : AIM, CARE, FAO, FIDA, GRET, ICCO, PAM, Welthungerhilfe. The program is in its initial phase. Period : 2013 -­‐ 2015 with a budget ± € 13 million (EU funding). ICCO participates in the consortium in collaboration with the local partner SAF / FJKM. 18 Projet d’Appui à la Maitrise d’Ouvrage Local pour l’Eau et l’Assainissement: Consortium of 3 Malagasy NGO (SAF/FJKM, Fikrifama et FSG) and ICCO. Period : 2007 – 2013.

Annex 4 – Report of the second phase of the Evaluation Programmatic Approach – Survey about the development and performance of coalitions

51

Annex 5 – Terms of Reference Advancement of ICCO Alliance’s working with the Programmatic Approach 2009 – 2012, phase 3, field study

Introduction This document provides the ToR for the field study phase of the Evaluation Programmatic Approach.

In the ToR of this evaluation it was envisaged that an eventual field study had to be justified by the evaluators and to be based on the findings of the first phase of the evaluation; the desk study and the online survey.

The desk study provided a comprehensive research into all documents written on the development of and the experiences of ICCO Cooperation on working with the Programmatic Approach. The findings of the documents were verified and deepened in interviews with the MT of ICCO Cooperation, programme officers and the coordinator Capacity Development.

The results19 of the research were presented and shared with the MT of ICCO Cooperation (September 2013). There was a consensus on the main findings with regards to the core elements of the Programmatic Approach in practice (the so-­‐called pearls of IA practice). Phase two of the evaluation was done through a survey that was sent to Programme Officers involved with coalitions and representatives of the coalitions. The survey aimed at creating more clarity about the developmental aspects of coalitions and the programmes they implement (and answers some of the evaluation questions on these topics).

Findings phase one The initial synthesis report defines besides a notion of the need to come up with clarity on the concept ‘programmatic approach’, and the need for more leadership on the programmatic approach in the future, 6 elements of the programmatic approach which are to be considered its essence and need to find their way into future formulation of the ICCO Cooperation’s way of working on the programmatic approach. The elements are shortly (see annex 1 for a more elaborate overview of the pearls): ownership, local dynamics, partnership, trust & accountability, joint action, learning and flexibility.20

Findings phase two The findings from the survey21 have largely confirmed the findings from phase one. More than fifty per cent of the survey respondents were positive about ownership within coalitions/ programs, feeling of responsibility for the cooperation, the coalition’s’ implementation capacity, integration of gender equality and rights-­‐based approach and effects of the cooperation. Less confidence is reported in relation to financial sustainability and fund diversification.

The findings of the survey have also been used to gain an insight in which coalitions score high on the survey elements that are comparable to/have relevance for the ‘pearls’ and can therefore be seen as proxy for achievement on the pearls. The thinking behind this is that if coalitions score high on these survey elements there is something to learn from that experience about the conditions that made this possible. The choice of coalitions to be included in phase 3 should be as much as possible within this range of high scoring coalitions.

Objective phase 3 Based on the initial findings, it seems to be relevant to design the field visits of the evaluation PA in such a way as to get a deeper insight into the practice of the formulated pearls.

19

Evaluation Programmatic Approach IA. Initial synthesis of findings, September 2013.

PREZI Kees Zevenbergen for MT October 2013 21 Evaluation Programmatic Approach, Survey about the development and performance of coalition. Verona Groverman and Kees Zevenbergen, 9 October 2013 20

Annex 5 – ToR field study

52

The purpose of the three field studies is to unearth the lessons learned from three coalitions that are considered to be successful and to formulate guidance for the further development of the Programmatic Approach (seen through the lenses of the 6 already identified ‘ pearls’).

Main objective for the field studies would therefore be to shed more light on the questions of why, how and what exactly a number of strong performing coalitions have done to become strong performers, including an analysis of the circumstances and/or preconditions that might have contributed to their successful emergence (including the roles that ICCO Cooperation staff played therein as ‘backbone support organization’22).

The strong performers will be selected on the basis of findings of the survey about the development and performance of coalitions23 (phase 2 of the PE Programmatic Approach) and on what Regional Managers consider to be their best performing programmatic coalition in the region.

Main research questions The main research questions relate to the above mentioned six key elements of the programmatic approach, the ‘pearls’, which together are assumed to contribute to systemic change. Coalitions, i.e. groups of organisations and institutions that purposely join forces to collectively address a certain key problematic (‘a big issue’), could be considered as ‘instruments’ to achieve such change. In the field study, the coalitions and its members are the subject of study through which deepened insight into the PA will be gained. We make a distinction between members of a coalition and stakeholders, considering the latter as a broader group of actors that are not necessarily involved in the full process to address a key problematic (starting from a joint analysis to monitoring and reflection).

During the field study primary attention will be on the coalition and its members and there where relevant other stakeholders will be included.

During the field study, the researchers will address the following general questions for all pearls distinguished: • To what degree has the pearl concerned indeed materialised by or in the specific context of the coalition?

• What concretely have the members of a coalition done in this respect, how have they done it and when did they do it?

• Who of the coalition’s members have been involved in the process, what roles has each of them played and how did they work together?

• What factors in particular helped to make the process a success?

• If the pearl concerned appears not to have materialised to a significant degree, how has the coalition addressed the issues at stake (‘how has it worked around the issues, or, what alternative path has been followed)? Are there any specific conditions to be indicated for this?

Apart from the general questions above that will be addressed for all pearls24, the following specific questions will be addressed as well for a number of pearls.

22

See John Kania & Mark Kramer (2011) Collective impact. Standford Social Innovation Review. Winter 2011, where reference is made to five conditions of successful collective impact in initiatives, i.e. a common agenda, shared measurement systems, mutually reinforcing activities, continuous communication, and backbone support organizations.

23 Strong performers are programmatic coalitions who -­‐ according to answers on given on the survey -­‐ perform well to excellent on 3-­‐5 of the pearls.

24 Concerning pearl 1 (for change processes to be relevant, they are to be grounded in a thorough and continued understanding of the ever-­‐changing context, issues at stake, power relations amongst the major actors and the dynamics on how these actors relate to one another and to the systems of which they are part) Annex 5 – ToR field study

53

Pearl 2: for change to happen, working at multiple levels with multiple stakeholders is necessary.

• Is the full ARE IN25 configuration included in the coalition?

• Are other stakeholders included in the coalition’s activities and if so, on which issues do they cooperate and how are the working relations maintained?

• Is the coalition and, where applicable, the multi-­‐stakeholder configuration composed of the right players to address the problematic in its systemic character?

Pearl 3: for stakeholders to indeed join forces effectively and enter into some form of joint and emergent action, a shared vision on the change and transformation to be promoted and about how to go about that (in other words a mutual consensus on a ‘theory of change’) seems crucial.

• To what degree did the strategic planning process result in a clear agenda of action? Does the agenda concern the coalition as a whole or mostly/only the activities/ actions of specific members?

• To what degree did the strategic planning process result in clear roles of members of the coalition building on and reinforcing each other’s strengths?

Pearl 4: for change to be sustainable, it seems crucial to address ‘the big issues’, the systems that generate poverty and injustice. This pearl closely relates to previous pearls. The following questions are relevant to be addressed in the field study.

• To what degree does the agenda for action (pearl 3) effectively address the big issues (based on the contextual and institutional understanding -­‐ pearl 1) and what assumptions are underlying the relation between the two?

• To what degree does the ‘big issue’ guide the daily practice/ implementation of the activities/ action of the coalition?

Pearl 5: for change to be effective, learning on what happens, on what works and what does not and consequently flexibly adapting one’s set of activities is key.

• To what degree does the coalition have a shared measurement system?26 • How are single loop learning (are we doing things right), double loop learning (are we doing the right things) and/or triple loop learning (are we testing our assumptions about how change happens) taking place and how is this embedded in the daily practice?

Methods of data collection and analysis This part of the evaluation will be carried out in the form of a field study. The fieldwork will be carried out with 3 coalitions, and is designed to find answers on the main questions above.

The field research will be undertaken through analysis of existing documents (context analyses, theories of change, MoU’s, agenda’s and plans for action, PMEL systems) and through interviews with various stakeholders, focus group discussions, workshops bringing together coalition members, where possible drawing on network analyses and power analyses to complement the research.

Before the start of the field study, the evaluators will propose, in consultation with the coordinator Capacity Development and the PMEL unit a short list of program coalitions to be visited. Main criteria for selection are: -­‐ a high score on the pearls in the survey; -­‐ as much as possible matching the pearls mentioned by the RMs and pearl 6 (relations amongst the cooperating parties are based upon trust and mutual accountability) the general questions will be sufficient to gain a deepened insight.

25 ARE IN: those with Authority, Resources, Expertise, Information, Need/ Stake 26 See footnote 4.

Annex 5 – ToR field study

54

-­‐ not included in the action researches or -­‐ part of the first phase of the PE PA The coalition distribution in a country must be such that the evaluation can be done within its timeframe.

The division of the field studies among the consultants is the following: -­‐ Verona Groverman: 2 field visits -­‐ Kees Zevenbergen: 1 field visit The field visits will be carried out in the period November – beginning of December 2013

In the final analysis stage, after the field study, the various data and findings will be compared and analysed to answer the research questions.

Deliverables A concise report on the main findings of the field study, and an adapted overview of quality criteria for the 6 pearls, their assessment in practice and points for attention for working with the pearls.

Based on the 3 sub-­‐reports (phase 1, 2 and 3) a short summarising report is delivered, in which the questions of the ToR are answered, recommendations to ICCO Cooperation are given as well overall lessons learned . The emphasis of the overall report will be on the recommendations to ICCO Cooperation regarding how to define the Programmatic Approach in a conceptually clear manner (including proposing a new ‘name’), and detailing what the preconditions that need to be fulfilled to be able to promote as ICCO Cooperation the flourishing of the ‘pearls’. If in the context of the evaluation (a) new pearl(s) is (are) identified these need to be included in the report. In this sense the summarizing report needs to have a ‘formative’ character.

This summarizing report will be delivered before December 31, 2013.

Annex 5 – ToR field study

55

Annex -­‐ Pearls of the IA Practice – a first attempt at quality criteria

A number of elements of the programmatic approach are however much cherished by most people interviewed, and indeed often mentioned in explaining the IA way of working to partners and outsiders alike. These elements are considered to constitute the ‘pearls of the IA practice’ that are most definitely to find their way into the future formulations of the IA way of working:

Pearls of Practice What is needed?

How to assess? for change processes to be • Context analysis • Regular appreciation of relevant, they are to be grounded (geography and theme availability, quality and in a thorough and continued specific) relevance of analyses understanding of the ever-­‐ • Problem tree changing context, issues at stake, • Actor constellation map power relations amongst the • Drivers of Change major actors and the dynamics analysis on how these actors relate to one • Political Economy another and to the systems of analysis which they are part; • Conflict Transformation analysis • Grassroots Democratisation analysis • Power analysis • Value Network Analysis • All regularly updated and shared by all concerned for change to happen, working at • Actor constellation map • Regular appreciation multiple levels with multiple whether all actors • Value Network Analysis 27 stakeholders is necessary; relevant for change are • Use ARE IN principle engaged in the agenda for action for stakeholders to indeed join forces effectively and enter into some form of joint and emergent action, a shared vision on the change and transformation to be promoted and about how to go about that (in other words a mutual consensus on a ‘theory of change’) seems crucial; for change to be sustainable, is seems crucial to address ‘the big issues’, the systems that generate poverty and injustice;

Theory of Change Agenda for action Consensus thereon by parties concerned

•

Regular appreciation of quality of ToC and of effective engagement thereof by parties concerned

Convincing story on how the agenda for action effectively addresses and changes the systems…

• Using methodology that allows for/ promotes emergence of systemic changes often marked by complexity

•

Pitches accepted at decision-­‐making bodies Strategies developed regularly updated and ‘validity’ tested Assumptions explicit and used in PMEL

• • •

•

•

•

27

Those with Authority, Resources, Expertise, Information, Need (stake) are welcomed and participate actively.

Annex 5 – ToR field study

56

Cynefin framework used to determine strategy Four Quadrants framework used Emergence sought.

•

•

for change to be effective, learning on what happens, on what works and what does not and consequently flexibly adapting one’s set of activities is key; relations amongst the cooperating parties are based upon trust and mutual accountability.

• • • Learning agenda and investments • Monitoring systems operational • Flexible and rolling planning and budgeting logics •

• • • •

Horizontal and downward accountability mechanisms Peer to peer exchanges Transparent governance set-­‐up Use ARE IN principle28 ICCO is steward in trust building, relations building and mutual accountability

•

•

•

Regular appreciation of quality of PMEL systems (including the effective use made thereof) Theory of Change and assumptions incorporated in PMEL Regular appreciation of quality of cooperation through self –assessment using Pscan

Key issue is on the ownership of the change processes. It can be argued that the owner(s) of these processes are the ones that determine the quality criteria and that are to uphold these. In other words: the owners of the change processes are to regularly appreciate to what degree the various elements of ‘good practice’ are present and up to standards.

28

Those with Authority, Resources, Expertise, Information, Need (stake) are welcomed and participate actively.

Annex 5 – ToR field study

57

Annex 6 -­‐ Report of the field (third) phase of the Evaluation Programmatic Approach

Evaluation Programmatic Approach IA FINDINGS FIELD PHASE INTRODUCTION

This paper presents the findings of the field phase of the evaluation of the programmatic approach as shaped and implemented by the ICCO Alliance (IA).

The outcomes of the first phase of the evaluation (a phase during which the evaluators read and analysed all material available on the launch, emergence, and implementation of the programmatic approach, including material on the subsequent learning thereon, and during which additional data was gathered in dialogue with key players in the ICCO Alliance) were laid down in a document entitled “Evaluation Programmatic Approach IA – Initial Synthesis of Findings” (September 2013).

These findings were presented to and discussed with IA leadership during their September MT meeting.

The research and dialogues revealed that a number of elements of the programmatic approach are much cherished by most, and indeed often mentioned in explaining the IA way of working to partners and outsiders alike. These elements are considered to constitute the ‘pearls of the IA practice’. These ‘pearls’ are shortly on: • anchoring interventions in context and local dynamics; • working with multiple actors at multiple levels; • sharing a vision on change and how to achieve that; • addressing the big issues; • learning and adapting; and on • creating trust and mutual accountability.

INTRODUCING THE FIELD PHASE OF THE RESEARCH The purpose of the field studies was to unearth the lessons learned from three coalitions that are considered to be interesting and/or successful and to formulate guidance for the further development of the Programmatic Approach. The main objective for the field studies was to shed more light on the questions of why, how and what exactly a number of coalitions have done to become strong performers, including an analysis of the circumstances and/or preconditions that might have contributed to their successful emergence (including the roles that IA staff played therein as ‘backbone support organisation’).

During the field study, the researchers addressed the following general questions for all pearls distinguished: • To what degree have the six pearls indeed materialised in the specific context of the coalition?

• What concretely have the members of a coalition done in this respect, how have they done it and when did they do it?

• Who of the coalition’s members have been involved in the process, what roles has each of them played and how did they work together?

• What factors in particular helped to make the process a success?

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

58

•

If a pearl appears not to have materialised to a significant degree, how has the coalition addressed the issues at stake (‘how has it worked around the issues, or, what alternative path has been followed)? Are there any specific conditions to be indicated for this?

In brief: the purpose of the field research was to better understand and document the conditions and factors that contributed to the emergence of successful coalitions.

INTRODUCTION OF THE THREE COALITIONS The selection of the three coalitions to be visited was based upon an analysis of the results of the online survey and furthermore on what Regional Managers considered to be their best performing or most interesting coalitions in the region.29 During November and December 2013, the evaluators visited the following three coalitions: • in Vietnam: the Vietnam Rivers Network; • in Uganda: the Ugandan Health Programme; and • in Senegal: le Programme Multi Acteurs.

During their field work, the evaluators had meetings with all relevant staff of participating organisations, exchanged with the coalitions leadership, interviewed individual coalition’s members and some other stakeholders, and held workshops with all concerned to unearth the lessons that could be learned from the collaborating efforts of the organisations concerned.

Let us first introduce the three coalitions. This will help in understanding the various observations of the evaluators.

The Vietnam Rivers Network (VRN) – VRN works to protect the health of riverine communities and ecosystems. In November 2005 a few concerned individuals established VRN. VRN has by now evolved in an open forum whose membership includes NGOs, researchers, academics, government officials, local communities, and individuals, all concerned with protection of rivers and river-­‐ dependent communities and sustainable development in Vietnam.

In 2007/2008 VRN went through a process of strategy development resulting in VRN’s Strategy 2008-­‐ 2020, based on which annual action plans are made and results reviewed in annual meetings open to all members. Its activities include sharing information, conducting research on social and environmental impacts caused by hydropower projects and other water-­‐related development projects in Vietnam, and doing advocacy on these issues. VRN has been successful in advocacy in which VRN members, communities and other local stakeholders are actively involved. Through its careful, transparent and evidence based approach, VRN has built up reputation and government recognition.

VRN has set up a structure with a management team, composed of three key organisations (each operating in one of the regions of Vietnam) and a secretariat, taken up by one of the three key organisations on a rotational basis. A third important entity in the structure is the advisory board composed of (active) individual members.

The three key organisations provide an important link with the communities. Their programmes/projects focus on awareness raising, building capacities on issues related to water governance, and empowerment of communities which helps male and female community members to voice their concerns and make their voices heard through the linkages that the organisations establish with authorities at different levels. At the same time they make authorities more conscious of the fact that community members have important issues to bring forward.

ICCO has been one of the main funders of the VRN evidence-­‐based advocacy programme since 2009.

VRN has another core funder and a few other donors for specific activities. Funds provided by IA are 29

Some very likely candidates were not included in the sample as they had participated in learning exercises or action researches, or as they were evaluated recently, or because visiting them was logistically not possible.

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

59

used for capacity building of VRN members; empowerment of affected communities through rights promotion, funding and setting up the community-­‐based river monitoring; research to create evidence for advocacy work; advocacy related to laws on water resources and downstream impacts of hydropower dams. The role of the IA programme officer is mainly facilitating with some brokering.

In terms of decision-­‐making, VRN is totally independent from any donor.

The Ugandan Health Programme (UHP) – UHP is a coalition of partners of four IA members (Tear, TWR, Woord en Daad, ICCO). In 2011, stimulated by the IA, the partners joined forces to implement and monitor a MFSII–funded IA thematic country programme related to health/HIV&AIDS. The programme’s two objectives are: 1) Well-­‐established accountability mechanisms in which civil society effectively calls the health system to account for the delivery of equally accessible basic health care; and 2) Capacitated change agents through which civil society promotes effective prevention of SRH problems, HIV transmission and disabilities. The UHP has a strong funding relation with IA, the IA being its main funder. The IA regional office also plays an important role in brokering and supporting capacity development. The members of UHP feel a strong ownership of the coalition and take decisions independently.

The membership of UHP is ‘closed’ and consists of six ‘like-­‐minded’ NGOs mainly working on grass roots level in the North and North East of Uganda. Five of them have been involved in the establishment and development of the UHP and the sixth member, focusing on advocacy, joined rather recently.

UHP members implement programme activities that are in line with their own mandate, approach and geographical area. These fit nicely with the IA ambitions in health. Apart from these projects, the members engage in joint UHP activities: capacity building of the members, sharing of experiences during meetings, exchange visits to each other’s projects, and collaborating in joint events. UHP does not jointly engage in advocacy; presently one member organises an advocacy campaign and others join in.

UHP has a light structure: a (paid) coordinator, steering group, and financial controller. The members hold (bi-­‐annual) meetings for review, sharing, reflection and planning of common activities.

The ‘Programme Multi Acteurs’ (PMA Senegal) – PMA is an IA-­‐financed programme implemented by five Senegalese NGOs. ICCO has financed activities of individual Senegalese NGOs since the 80’s of the last century. These activities were mainly in the field of education and literacy. The introduction of the programmatic approach has led the IA to actively stimulate the collaboration between the Senegalese NGOs, which has resulted in the creation of a ‘Consortium Multi Acteurs’ in 2009. This coalition, at the time composed of 13 Senegalese NGOs, launched a ‘Programme Multi Acteurs Education’ during 2009 (PMA-­‐1). The coalition entered a rather painful process of reformulating its activities and of reducing its membership as a consequence of budget reductions, some reported irregularities within a partner organisation and of ICCO’s choice to terminate its support to education programmes and to refocus on vocational training and, later on in the process, also on sustainable economic development through value chain development. This process resulted eventually in the current ‘Programme Multi Acteurs 2 Second Phase’ covering activities in non-­‐formal education, technical and vocational training, value chain development and sexual and reproductive health.

PMA-­‐2 is implemented by five Senegalese NGOs (of which two are national offices of international NGOs). The activities of the Consortium are presented in a single (multi-­‐) annual planning document.

Based on this document, the five individual organisations each present their activities in separate annual plans and budgets that are approved in bilateral contacts between the IA regional office in Bamako and the individual NGOs. The coalition is led by a ‘comité de pilotage’ and coordinating work is done by a focal point (one of the five NGOs) and a programme coordinator. The programme coordinator organises regular planning and monitoring meetings.

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

60

The IA staff of the regional office plays a very ‘hands on role’ both in determining the thematic direction of the activities undertaken, as well as in the individual planning and budget approval phase.

INITIAL OBSERVATIONS The three coalitions visited are very different in many a way: • genesis – the VRN is established by individuals driven by an endogenous will to unite and join forces, whereas the UHP and the PMA are formed on the instigation of the IA; • composition – the VRN is composed of a rich pallet of organisations and individuals (NGOs, research institutes, local communities, advocacy organisations, government officials), whereas the UHP and the PMA are composed of more or less comparable NGOs with which the individual IA members have had prior (often long-­‐standing) funding relations. The UHP has however recently been expanded with a dedicated advocacy outfit; • raison d’être – whilst the core of the VRN activities is on joint-­‐action (evidence-­‐based advocacy), the core of the UHP and PMA programme is on implementing mainly field-­‐based activities of the individual NGOs. (Indicative for this is already in the names of the coalitions: VRN being a network and UHP and PMA being programmes); • programming – linked to the above, the VRN programme presents only the activities to be undertaken by the network or for the network purposes, whilst the UHP and PMA programmes mainly consist of activities to be undertaken by the individual NGOs with only very limited attention for joint-­‐activities; • external orientation – VRN is very much oriented towards the outside world in realising its core-­‐ business (evidence-­‐based advocacy) and readily and easily engages with non-­‐member parties if it is in its interests. UHP and PMA are more internally oriented; • funding – the IA is the sole funder of the UHP and PMA programmes (although the individual partners do sometimes have other sources of funding for the other activities they undertake), creating a great dependency on the IA for both the implementation of the individual activities as well as of the continuation of the collaboration. VRN has a broad funding base for both its joint activities as well as for the activities of the individual members, increasing its autonomy and independency in decision-­‐making; • areas of expertise – the VRN and the UHP undertake activities in thematic areas in which they have experience and a solid track record; the programmes they implement concern their core business. The PMA, especially after its reorientation, is taking its partners in thematic areas in which they do not have any (or very limited) experience or track record (especially with regards to business development services, value chain development and support to economic activities); • IA support – IA support to the VRN mainly consists of moral and financial support, whereas IA in Uganda also provides for active in situ tailored capacity building activities and facilitating support to UHP, and, to a minor extent, brokering. IA support to PMA in Senegal mainly consists of financial support, more general capacity building support through trainings in Bamako and through specific guidance on the strategic direction of the programme; • visibility of the coalition – VRN has a very clear online profile and mentions ICCO as one of its donors. The UHP is invisible online. The PMA is equally invisible online. The five participating NGOs do not mention the PMA on their websites and only few mention ICCO as a donor. The IA

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

61

•

website does not mention PMA and it states that ‘The ICCO Cooperation runs an Education program in Senegal’. business opportunities for IA – from a more business-­‐oriented perspective, the picture is equally diverse. The issue is to what extent the IA can raise its profile or increase its fundraising potential through support to either one of the three coalitions. For VRN, the IA role as a funder is important, but, as the VRN activities focus on water security and advocacy on environmental issues, profiling on the major IA themes does not seem directly obvious. Rebranding might improve the situation. For PMA, the current thematic fit seems better, although one might raise the question whether the reorientation of the programme does not call for a reorientation to a partner-­‐set better equipped for intervening in value chains. For UHP, IA business opportunities seem best: coalition of trusted partners, thematic fit, well appreciated role of IA staff. Challenge here is to profile the IA as a provider of crucial and valued capacitating, brokering, and facilitating services that merits further support in the future.

The evaluators have not analysed the degree to which the three coalitions produce their outputs and achieve their outcomes. They are therefor not able to relate the abovementioned diversity to any significant differences in the effectiveness of their operations. The evaluators have however analysed the various coalitions on the degree to which they relate to the various key-­‐elements of the IA programmatic approach, the ‘pearls of the IA practice’. Such under the assumption that the better that one scores on these pearls, the more likely it is that the coalitions are successful in their endeavours in pursuing and eventually realising sustainable system’s change.

MORE DETAILED ANALYSIS ON THE PEARLS OF THE IA PRACTICE Pearl 1: For change processes to be relevant, they are to be grounded in a thorough and continued understanding of the ever-­‐changing context, issues at stake, power relations amongst the major actors and the dynamics on how these actors relate to one another and to the systems of which they are part.

All coalitions have in one way or another tried to analyse the context in which they operate and define and implement their activities. Quality, completeness and timeliness however vary.

Positive developments: • all realise that thorough understanding of context is very important to shape and implement relevant and effective interventions; • all invest time and resources in acquiring that understanding; • more thorough analyses are made when results of researches (often executed by professional researchers) are discussed and analysed during (multi-­‐stakeholder) workshops with at least the participation of the key-­‐players of the organisations engaged in the coalition; • the better analyses are made when they zoom in from a more generic level to a well defined and specific geographic or thematic area.

Issues for further reflexion: • most context analyses are produced in function of the formulation of programmes to be submitted to donors. They are often a chapter in a document that is only updated when a new proposal is to be submitted; • most context analyses are situation or problem oriented. They sketch the nature and the magnitude of the problems in a given area and situation. Rare are the analyses that ask why questions and that unearth the underlying root causes of the problems indicated and/or the Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

62

•

underlying power dynamics that either contribute in perpetuating the problems indicated or that could be mustered and harnessed to stimulate change processes to sustainably deal with these root causes; tools and methods for solid and repeated contextual research and analysis indicated in the IA guidance note on the programmatic approach are seldom used.

Pearl 2: For change to happen, working at multiple levels with multiple stakeholders is necessary.

All coalitions work at multiple levels with multiple stakeholders, although at varying degree. The extent to which coalitions include those with authority, resources, expertise, information and need or stake (ARE IN) in their actions however varies.

Positive developments: • all partners currently active in the three coalitions come from a background where they went about their business more or less alone. In most coalitions, the IA has played a valuable and cherished role in bringing them together and in supporting them in shaping their collaboration; • most valued in multi-­‐stakeholder settings are the opportunities to generate more clout in achieving ones objectives and to learn from one another; • most of the coalitions have emerged from a stakeholder analysis during the start-­‐up phase, which however hardly included an analysis of the power dynamics between actors; • the better coalitions are those that indeed include those with ARE IN. Even better coalitions are those that not only include ARE IN, but ARE IN at the various levels needed to generate effective and lasting change; • some coalitions have achieved results the individual members could have never achieved alone, especially with regards to effective evidence-­‐based advocacy in which real life stories and hard data are coupled to community mobilisation, top-­‐end communication and policy influencing up to the highest levels; • successful multi-­‐level multi-­‐actor collaboration depends more on highly engaged and motivated individuals than on the formal engagements between organisations.

Issues for further reflexion: • most coalitions are composed of comparable, like-­‐minded, NGOs that do more or less the same thing in different geographical areas. Few are the coalitions that join actors that truly add value to one another on their typical area of expertise in a concerted way on a specific change process; • most actors in coalitions do things or the type of things that they used to do before and invest only very limited resources in joint-­‐action or actions for which they truly need each other; • joint-­‐action is in general more focussed on strengthening the capacities of the coalition’s members than on system-­‐change oriented coalition activities; • most coalitions, especially the ones that are created on instigation of the IA, are not composed of those partners that really need each other to launch successful change processes, but are mainly composed of partners with which the IA had a prior funding relation; • most coalitions could generate more success when investing more on evidence-­‐based advocacy activities either themselves or through including experienced researchers, communicators and lobbyists in their coalitions.

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

63

Pearl 3: For stakeholders to indeed join forces effectively and enter into some form of joint and emergent action, a shared vision on the change and transformation to be promoted and about how to go about that (in other words a mutual consensus on a ‘theory of change’) seems crucial.

All coalitions have developed a multi-­‐annual (strategy) plan, often including a vision and mission statement. The degree to which the rationale for joint-­‐action is developed and the degree to which pathways towards change and transformation are identified (either written or in spirit) however vary.

Positive developments: • all coalitions have engaged in consultative and collaborative efforts to develop (multi-­‐annual) planning documents; • most activities are geared to uplifting the situation of poor and vulnerable people; • the better coalitions do not only have a shared vision on the change and transformation to be promoted, but have also clearly indicated why and how the participating organisations need each other to effectively support those and realise sustainable change. Or, to rephrase; • the better coalitions focus more on concerted and joint-­‐action to generate change than on the various activities of the individual organisations; • the ‘theory of change approach’ (ToC) is generally considered to be a very useful approach in focusing the participating organisations on the real issues at stake, on the best ways and means to jointly go about them and on who could do what best to generate sustainable results.

Issues for further reflexion: • the ToC is still in its infancy stages. Although the initial trainings on the ToC have generated interest and enthusiasm, in situ support to coalitions in actually developing a tailored ToC was solicited in most cases and will most definitely strengthen them in developing a shared vision on the change and transformation to be promoted and on how to go about that; • it appears to be of key-­‐importance that this support is well facilitated (and that is probably best provided by external facilitators) as the process of developing a ToC might reveal internal weaknesses in (the composition of) various coalitions, and/or reveal ownership issues; • although gender is considered a cross-­‐cutting issue by all, only members working at the community level address gender or women’s issues. The higher-­‐level agenda’s for joint-­‐action do not reflect commitment to gender equality.

Pearl 4: For change to be sustainable, it seems crucial to address ‘the big issues’, the systems that generate poverty and injustice.

This is a tough one! All coalitions do indeed address the big issues that complicate the lives of poor and vulnerable people (access to health, water, resources, economic opportunities, etc.). Rare are however the coalitions that address their underlying causes, the systems that generate poverty and injustice.

Positive developments: • the initial work done with the ToC helps to get better insights in what the big issues are and which their systematic dimensions are; • evidence-­‐based advocacy seems to provide for very promising (and probably lasting) opportunities to generate systems change (for example: empowered communities able to stop dam construction, proof that joint-­‐action can indeed make a difference, laws that are modified to better protect the environment and interests of communities).

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

64

Issues for further reflexion: • understanding the difference between ‘addressing a problem’ and ‘addressing its root-­‐causes’ remains a challenge. For many concerned working in for example HIV/AIDS is addressing a big and systemic issue. Whilst for them working on one of its root-­‐causes, for example working on why especially young men are increasingly promiscuous (and working on related masculinity-­‐ issues) is far less likely; • addressing systems that generate poverty and injustice is a highly politicised and often risky endeavour that does not only require solid power analyses (that are often lacking as indicated above), but also the time to make a difference. The current short-­‐term and results focused IA contracts might not favour the aspired systems change; • more effort might be needed to align with and strengthen existing government structures that are mandated by policy frameworks. NGOs might be more effective by being more transparent and open to promote willingness at government agencies.

Pearl 5: For change to be effective, learning on what happens, on what works and what does not and consequently flexibly adapting one’s set of activities is key.

All concerned realise that learning is of key-­‐importance, not only because it contributes to becoming effective and efficient agents of change, but also because the process of learning itself enhances the quality of relations and interactions within a coalition. The quality of learning however varies.

Positive developments: • all coalitions organise (bi-­‐)annual meetings to review outputs and outcomes of the annual plans with a focus on ‘are we doing things right’; • factors that help to make the process of learning (through reviewing, (documented) peer learning and exchange visits) and adapting a success are the active participation of the cooperating parties, and the inclusion not only of leadership, but especially of the thematic specialists of the organisations concerned.

Issues for further reflexion: • monitoring systems and tools are often of poor quality and generate more output data that outcome information; • the monitoring efforts are focused more on results generated by the individual organisations than on results of the coalition; • for learning to happen, one needs more information than on outputs and outcomes only.

Learning also requires process-­‐information. ‘How did we go about in realising this result?’ This is especially true for delicate processes of successful advocacy which are currently hardly documented and/or analysed; • very little double and triple loop learning takes place. Questions like ‘are we doing the right things’ and ‘how do we know right from wrong’ are seldom asked. Apart from the fact that these are tough questions, people and their organisations are often so occupied by the daily matters at hand (and the often very heavy planning and reporting formats) that they just don’t have or take the time for them.

Pearl 6: Relations amongst the cooperating parties are based upon trust and mutual accountability.

Yes, the better these relations are, the more likely it is that the above-­‐mentioned pearls do indeed start to shine.

Positive developments: Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

65

• •

all coalitions have invested in relation building and mutual accountability; important ‘instruments’ to promote trust and mutual accountability are regular communication of updates by the coordinator, the regular bi-­‐annual review meetings in which discussion take place – people agree, disagree, and respect each other, peer-­‐to-­‐peer exchanges, a clear division of roles and responsibilities among partners, involvement of leadership in the coalition’s activities, shared leadership in the coalition, and a felt and realised ownership over the coalition’s direction.

Issues for further reflexion: • In cases in which coalitions are still dealing with the fall-­‐out of the recent reorientation and reshuffling of membership as promoted by IA, they are more of a marriage of convenience than based upon genuine trust and respect; • Transparent, participatory and decentralised processes of budget allocation and approval increase the quality of the relations amongst the cooperating parties.

REVIEWING THE PEARLS OF THE IA PRACTICE… Reviewing the three coalitions through the lens of the six pearls proved not only to be a feasible process, but also to be a highly interesting process as it clarified major strengths and weaknesses of the coalitions and the areas on which they could work to improve and strengthen their operations and increase their shot at effectiveness.

There are however two areas that seem not to be sufficiently covered by the six pearls. And that are the areas that have to do with: • the ownership over and quality of the internal organisation of the coalition, dealing with issues like decision-­‐making over strategic direction, planning and budgeting, communications, coalition-­‐wide PMEL and financial control, capacity building and institutional, social, economic and financial sustainability; • the nature and quality of the support provided by the backbone support organisation, in casu by the IA. Both areas are, as was confirmed during the three field visits, like the earlier mentioned pearls, of critical importance in generating and supporting sustainable change processes.

The field visits have also confirmed the validity of the findings and initial recommendations of the first phases of the current evaluation. In the concluding paper, these will be enriched with a number of lessons learned during the field visits and a number of final recommendations towards the IA on how to further strengthen the programmatic approach of the alliance.

Verona Groverman, Kees Zevenbergen – February 2014

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

66

Annex A -­‐ documents reviewed field visits

Literature -­‐ Uganda - Thematic Country Plan 2011-­‐2015, Uganda – Health and HIV & AIDS -­‐ Uganda Country Plan 2013, undated - Country year plan 2013 (May 2013, updated version) - Ochen, Richard & Janharmen Drost. Program Progress Report 2012 Uganda Country Health and HIV/AIDS programme. January 2013 - Proposed action IA-­‐ Uganda Health Program. Global Face-­‐to-­‐face meeting related to the Basic health, SRHR & HIV program of the ICCO Alliance. 24-­‐26 June 2013, Soest, the Netherlands - ICCO-­‐ALLIANCE UGANDA HEALTH PROGRAM LEARNING AGENDA, undated - Invitation for UMOJA training workshop for the Uganda Health Partner Programme -­‐ ICCO Alliance and ICCO Alliance Uganda Health & HIV Cluster AIDE MEMOIR).

Literature – Vietnam About the network: - VIETNAM RIVERS NETWORK (VRN). VRN STRATEGY 2008 -­‐2020. Hanoi, January 2009 (including annexes) - VRN’s 2011 Annual Report. Undated (reporting concerning the VRN STRATEGY 2008 -­‐2020) Concerning ICCO support: - The Center for Water Resources Conservation and Development (WARECOD). Final proposal to ICCO 2011 – 2013. Capacity strengthening of the Vietnam Rivers Network: Advocacy and Research on rivers and sustainable use of water resources in Vietnam, including Annex 1 – 9.

Hanoi, Vietnam. January 2011 - Annex 1 Annex 1 VRN member list update in January 2011 (in Vietnamese) - Annex 2 Annex 2 VRN strategy and Action Plan - Annex 3 Funding sources for VRN action plan 2011-­‐2013 - Annex 4 Vietnam's Water Resource Challenges from Nancy - Annex 5-­‐Community based rivers monitoring and protection - Annex 6 VRN operation regulation – EN - VNR Annex 7 ICCO project timeline for 2011-­‐2013 - VNR Annex 8 and 9 Estimated budget for ICCO project 2011-­‐2013 - Project Plan. VRN Program 2011-­‐2014 Rivers and Watershed Management (76-­‐03-­‐02-­‐037). 27 May 2011.

-

Specific c ontract terms a nd c onditions. V RN P rogram 2 011-­‐2014 R ivers a nd W atershed Management (76-­‐03-­‐02-­‐037)

-

WARECOD. Narrative Report 2012. Capacity strengthening of the Vietnam Rivers Network: Advocacy and Research on rivers and sustainable use of water resources in Vietnam. 28 Feb. 2013 (almost the same report as WARECOD. Capacity strengthening of the Vietnam Rivers Network: Advocacy and Research on rivers and sustainable use of water resources in Vietnam Annual project update 2012. January 12th2013) Le Hien. Feedback on progress reporting 2012-­‐ Financial report, Narrative report, Auditor’s report. 09-­‐07-­‐2013

-

-

The Center for Water Resources Conservation and Development (WARECOD). Final proposal to ICCO. Promoting River Protection through Local Action, Linking and Advocacy (LALA for Rivers) -­‐ January 2014 to December 2015, including annex. Hanoi. 31 July 2013.

Project p lan – C onsiderations V RN 2 014-­‐2015 K eeping P eople S afe from W ater R eservoirs.

23 A ugust 2 013 Project Plan. VRN 2014-­‐2015 Keeping People Safe from Water Reservoirs (76-­‐03-­‐02-­‐051). 24 October 2013

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

67

Other reports - WARECOD FINANCIAL MANAGEMENT DEVELOPMENTS-­‐ 2012 - Strength Civil Society Sector. QUESTIONNAIRE ON THE STRENGTH OF CIVIL SOCIETY – WARECOD RESPONSE. January 2012 - Paul McCarthy & Jude Rand. BASELINE CAPACITY ASSESSMENT OF CONSORTIUM PARTNERS. OXFAM NEDERLAND / WWF GREATER MEKONG PROGRAMME, COMMUNITY ENGAGEMENT IN GOVERNANCE OF MEKONG AND SALWEEN RIVER BASINS . August 23, 2013 - VRN Overview agenda PA -­‐ TOC September 2013 and a few powerpoints from Saskia van Drunen (trainer) Literature – Senegal - Programme Multi Acteurs 2 second phase, juillet 2013 - AEA – document projet PMA 2, août 2012 - AEA – rapport technique 2012 - CERFLA – programme d’appui, août 2012 - CERLFA – note filière lait - CERFLA – budget opérationnel an 1 – PMA2 - CERFLA – rapport annuel, janvier 2013 - ONG 3D – Cadrage de la préparation de la seconde phase du PMA, 2012 - ONG 3D – Projet DRSP An 2, octobre 2013 - THP – rapport annuel 2012 - THP – project plan 2013-­‐2014 - USE – PMA USE 2012-­‐2015 - USE – rapport d’activités 2012 PMA II USE - USE – cadre logique de l’intervention du PMA2 USE

Annex – Time schedule and people met during field visits

Uganda Date and place Activities in Uganda 20/11 workshop -­‐ participants: ACET Office Paul Kabunga, Director ACET (partner van Tear) Makindye, Sarah C. Nampindo , Program Manager TAIP (partner van Tear) Kampala Martin Kizito, Program Officer TAIP James Peter Olupot, Executive Director PAG Kidep (partner van Woord en Daad) Richard Ochen, Program Manager HNU (partner van ICCO) Carol .., Trans World Radio Kenya (partner Trans World Radio Nederland) Samuel M. Ogutu, General Manager UCAN (partner van Woord en Daad) Edward Atenu, Program Officer UCAN Interview with: Richard Ochen -­‐ Programme Manager, HealthNeed Uganda 21/11 Interviews with: IC – RO office Paul Kabunga, Director ACET Muyenga, Samuel M. Ogutu, General Manager UCAN Kampala (Interview with Director of UNERELA+ was cancelled) 22/11 Interviews with hotel in Kampala Taaka, Janepher -­‐ Programme officer programme DCA IC – RO office Janharmen Drost -­‐ Programme Officer Health ICCO Cooperation Regional Office Muyenga Central and Eastern Africa Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

68

23/11

Vietnam Date and place 28/11, WARECOD office, Hanoi

29/11 Windy Hotel, 23 Thong Phong lane, Ton Duc Thang, Ha Noi

30/11 morning 30/11 1/12 morning 1/12 afternoon

2/12 morning CSRD office, Hue

2/12 afternoon,

Data processing and report writing Departure in the evening

Activities in Vietnam Interviews with: Ms Dao Thi Viet Nga, Director of WARECOD, Coordinator VRN in the North Ms Duong Thu Hang, project officer WARECOD, fundraiser for VRN Ms Hoang Thi Tu Oanh, project officer WARECOD, communication officer VRN in the North Mr. Tien si Vu Ngoc Long, Director of Southern Institutive of Ecology, Director of CBD, Coordinator VRN in the South Ms. Nguy Thi Khanh, Director of GreenID, former VRN coordinator, currently advocacy expert of VRN, Mr. Dang Ngoc Quang, Director of RDSC, member of advisory board Ms. Truong Anh Thi, researcher at CBD, communication officer VRN in the South Dinner with: Ms. Ms Lâm Thị Thu Sửu, Director CSRD in Central Vietnam, Coordinator VRN nationally Workshop -­‐ participants: Ms Lâm Thị Thu Sửu, VRN’s Coordinator nationally Ms Dao Viet Nga, VRN Coordinator of the North Mr Tien si Vu Ngoc Long, Coordinator VRN in the South Ms Ngụy Thị Khanh, Advocacy expert VRN Ms Trương Anh Thơ, communication officer VRN in the South Ms Hoàng Thị Tú Oanh, communication officer VRN in the North Ms Dương Thu Hằng, fund raiser for VRN Dao Trong Tu, Hydropower expert, director of CEWARED member of advisory board Dao Trong Hung, human ecologist, member of advisory board Mr Dan Tiep Phuc, law expert, member of advisory board Mr Nguyen Tien Long, energy expert CEWARED, member of advisory board Mr Dang Ngoc Quang, member of advisory board Ms Lương Thị Trường, Human rights expert CSDMA, member of advisory board Interview with Mr. Le Anh Tuan, Vice director Dragon Institute – Can Tho University in the Mekong region, member of advisory board Data processing and report writing Travel to Hue Report writing Interview with Mr Le Hien – Programme Officer Vietnam ICCO -­‐ South East Asia and Pacific Regional Office Report writing Interviews with: Ms. Ms Lâm Thị Thu Sửu, Director CSRD in Central Vietnam, Coordinator VRN nationally Mr Tran Ba Quoc , project officer CSRD, supporting VRN Ms Pham Thi Dieu My, project officer CSRD , supporting VRN Ms Tran Chi Thoi, project officer CSRD. supporting VRN Financial arrangements with CSRD related to assignment Interviews with:

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

69

coffee shop Hue

2/12 evening 3/12 Senegal Date and place 17/11/2013 18/11, office ONG3D, Dakar

19/11 morning 19/11 afternoon 20/11 morning 20/11 afternoon 21/11 morning 21/11 afternoon 22/11 morning 21/11 afternoon 21/11 night

Mr Nguyen Van Que, Union of Science and Technology, Social Impact Assessment, member of VRN Ms Le Thi Nhu Nguyen, Hue University, Social Impact Assessment, member of VRN Mr Pham Mau Tai, Director local NGO Quang Bing province, member of VRN Travel back to Hanoi Departure from Vietnam

Activities in Senegal Travel Amsterdam -­‐ Dakar Meeting with Comité de Pilotage of the PMA coalition. Present: Mme. Cissé, director of THP, president of the CdP Mr. Diallo, director of AeA Mme. Loune, secrétaire exécutive CERFLA Mr. Cissé, coordinateur ONG3D Mr. Diop, director of USE Mme. Diallo, consultante ICCO Mr. Moctar, coordinateur PMA Meeting with Aide et Action director and staff Meeting with The Hunger Project director and staff Meeting with CERFLA director and staff Meeting with USE Project director and staff Meeting with ONG3D director and staff Meeting with PMA2 Coordinator, preparation debriefing Debriefing and dialogue with PMA Comité de Pilotage Meeting with PMA2 Coordinator, finalisation of the mission Travel Dakar -­‐ Amsterdam

Annex 6 – Report of the third (field) phase of the Evaluation Programmatic Approach

70