GEF Annual Performance Report 2007 by GEF Independent Evaluation Office

GEF Annual Performance Report 2007 october 2008

Evaluation Office

G LOBAL ENVIRONMENT FACILITY

Global Environment Facility Evaluation Office

GEF Annual Performance Report 2007 October 2008 (The main findings and recommendations of this evaluation were presented to the GEF Council in April 2008.)

Evaluation Report No. 40

ÂŠ 2008 Global Environment Facility Evaluation Office 1818 H Street, NW Washington, DC 20433 Internet: www.gefeo.org Email: gefevaluation@thegef.org All rights reserved. The findings, interpretations, and conclusions expressed herein are those of the authors and do not necessarily reflect the views of the GEF Council or the governments they represent. The GEF Evaluation Office does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the GEF concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The GEF encourages dissemination of its work and will normally grant permission promptly. ISBN-10: 1-933992-16-6 ISBN-13: 978-1-933992-16-7 Credits Director of the GEF Evaluation Office: Robert D. van den Berg Task Manager: Aaron Zazueta, Senior Evaluation Officer, GEF Evaluation Office Evaluation Team: Neeraj Kumar Negi, Evaluation Officer; David Michael Todd, Senior Evaluation Officer; Anna Viggh, Evaluation Officer; Ines Angulo and Cynthia Cummis, consultants Editing and design: Nita Congress Printing: Graphic Communications Cover photo: Urucan seeds from Brazil, Igor Castro da Silva Braga, World Bank Evaluation Report No. 40 A FREE PUBLICATION

Contents Foreword....................................................................................................................................... vii Acknowledgments........................................................................................................................ ix Abbreviations................................................................................................................................. x 1. Background, Main Conclusions, and Recommendations..................................................... 1 1.1 Background....................................................................................................................................................... 1 1.2 Findings and Conclusions.............................................................................................................................. 2 1.3 Recommendation.......................................................................................................................................... 10 1.4 Issues for the Future...................................................................................................................................... 10 Notes......................................................................................................................................................................... 11

2. Scope and Methodology........................................................................................................ 12 2.1 Scope................................................................................................................................................................ 12 2.2 Methodology.................................................................................................................................................. 13

3. Project Outcomes and Sustainability................................................................................... 17 3.1 3.2 3.3 3.4

Rating Approach............................................................................................................................................ 17 Project Outcomes.......................................................................................................................................... 18 Project Outcomes Sustainability................................................................................................................ 21 Outcomes and Outcomes Sustainability.................................................................................................. 23

4. Factors Affecting Attainment of Project Results................................................................. 24 4.1 Materialization of Cofinancing.................................................................................................................. 24 4.2 Delays in Project Completion..................................................................................................................... 26 Note........................................................................................................................................................................... 29

5. Evaluation of Capacity Development .................................................................................. 30 5.1 Findings from the Country Case Studies................................................................................................. 30 5.2 Findings from the Terminal Evaluations.................................................................................................. 31 5.3 Summary......................................................................................................................................................... 33

6. Quality of Project Monitoring................................................................................................ 34 6.1 Rating Approach............................................................................................................................................ 34 6.2 Overall Findings............................................................................................................................................. 34 6.3 Factors Affecting Quality of Monitoring................................................................................................. 35

7. Quality of Terminal Evaluations............................................................................................ 37 7.1 Rating Approach............................................................................................................................................ 38 7.2 Findings............................................................................................................................................................ 38 7.3 Agency versus Evaluation Office Ratings................................................................................................ 41

8. Assessment of GEF Agency Climate Policies and Guidelines............................................. 42 8.1 Assessment Approach.................................................................................................................................. 42 8.2 Findings............................................................................................................................................................ 43

9. Management Action Record.................................................................................................. 44 9.1 Rating Approach............................................................................................................................................ 44 9.2 Findings............................................................................................................................................................ 45

10. Performance Matrix.............................................................................................................. 48 10.1 Rating Approach............................................................................................................................................ 48 10.2 Findings............................................................................................................................................................ 51

Annexes A. Terminal Evaluation ReportReview Guidelines....................................................................................... 53 B. Terminal Evaluations Reviewed during FY 2007...................................................................................... 59 C. Greenhouse Gas Footprint Policies and Guidelines Assessment......................................................... 61 D. Methodological Notes on the Performance Matrix................................................................................ 71 E. Management Response................................................................................................................................... 74

References.................................................................................................................................... 77 Figures 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 4.1 4.2

Percentage of Projects with Outcomes Rated Moderately Satisfactory or Above......................... 17 Percentage of Projects with Sustainability Rated Moderately Likely or Above.............................. 17 Number of Terminal Evaluation Reports by GEF Agency and Focal Area...................................... 18 Project Outcome Ratings by Number of Projects and Project Funding.......................................... 19 Percentage of Projects with Outcome Dimensions Rated Moderately Satisfactory or Above... 20 Sustainability Ratings by Number of Projects and Project Funding................................................. 22 Perceived Risks Underlying Projects Receiving Low Sustainability Ratings................................... 22 Projects with Outcomes/Sustainability Rated as Moderately Satisfactory/ Moderately Likely or Above........................................................................................................................ 23 Reported Materialization of Cofinancing per Dollar of Approved GEF Grant.............................. 25 Average Delay in Project Completion by GEF Agency and Focal Area, FYs 2005–07................. 27

GEF Annual Performance Report 2007

5.1 6.1 6.2 7.1 7.2 7.3 7.4 9.1

Types of Capacity Development Activities in Projects......................................................................... 32 Percentage of Projects Rated Moderately Satisfactory or Above for Quality of M&E ................ 34 Quality of M&E during Implementation................................................................................................. 35 Number of Terminal Evaluation Reports Submitted for Review....................................................... 37 Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above .............................................................................................................................................................. 39 Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above, by GEF Agency and Focal Area.................................................................................................... 40 Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above on Each Quality Dimension........................................................................................................... 40 Change in Ratings on Adoption of Council Decisions since APR 2006 Assessment.................... 46

Tables 1.1 Summary of Project Outcomes and Sustainability Ratings................................................................... 3 1.2 Materialization of Cofinancing.................................................................................................................... 5 1.3 M&E Ratings for Projects.............................................................................................................................. 6 1.4 Quality of Terminal Evaluation Reports.................................................................................................... 7 1.5 Ratings of GEF Progress in Adopting Council Decisions...................................................................... 9 3.1 Project Outcomes by GEF Agency and Focal Area............................................................................... 19 3.2 Outcome Ratings for FY 2007 by GEF Investment............................................................................... 20 3.3 Sustainability of Project Outcomes by GEF Agency and Focal Area................................................ 21 3.4 Sustainability of Outcome Ratings for FY 2007 by GEF Investment................................................ 22 4.1 Materialization of Cofinancing.................................................................................................................. 24 4.2 Materialization of Cofinancing by GEF Agency, 2002–07.................................................................. 26 6.1 Quality of M&E during Project Implementation by GEF Agency and Focal Area........................ 35 7.1 Quality of Terminal Evaluation Reports by GEF Agency and Focal Area....................................... 39 8.1 Assessment of GEF Agencies’ Climate Policies...................................................................................... 43 9.1 Ratings of GEF Progress in Adopting Council Decisions.................................................................... 45 10.1 GEF Agencies and Entities Performance Matrix................................................................................... 49

Contents

vii

Foreword The Evaluation Office of the Global Environment Facility is pleased to present its fourth annual performance report (APR) evaluating the performance of the GEF portfolio. Building and responding to work initiated in the three preced ing APRs, the evaluation based its conclusions on information collected by the monitoring and evaluation systems of the GEF Agencies.

Office. The 2007 APR also responds to a specific concern expressed in previous reports: the veracity of the terminal evaluation reports submitted so far had been checked through desk reviews only. To address this issue, the Evaluation Office piloted the approach to direct verification in the field of terminal evaluations described in this report.

The report describes results of an independent assessment of three topics highly relevant to the success of GEF efforts: project outcomes and sustainability, materialization of cofinancing, and quality of monitoring and evaluation arrangements. First and foremost, the Evaluation Office assessed the extent to which capacity development activities in GEF projects are relevant, effective, and efficient. The report also describes the results of these activities and their sustainability.

Findings presented in this APR show that the quality of terminal evaluation reports—on which the information and conclusions of this report so rely—has steadily improved. This can be attributed to efforts by the GEF Agencies and feedback provided to them by the Evaluation Office. However, the 2007 APR indicates that this fiscal year’s cohort has lower outcome and sustainability ratings than for previous years. It is too early to determine if this will constitute a trend, given the small sample size and differences in the typology of projects included in the 2007 cohort.

This year’s report goes further than its predecessors. With increasing attention worldwide to reducing greenhouse gas emissions, it is important that the GEF partnership address this issue in its own operations. This APR thus reviews the carbon footprint policies and guidelines of GEF entities and Agencies. In addition, for the first time, a performance matrix summarizes the performance of the GEF Agencies and the Secretariat on various parameters tracked by the Evaluation

The GEF Council reviewed the 2007 APR and its management response (see annex E) at its 33rd meeting in April 2008. The Council in particular congratulated the GEF Agencies on their progress in reducing greenhouse gas emissions in internal operations and invited them to provide information on progress in June 2009. Some Council members requested that the Evaluation

Office address cofinancing in detail in future evaluations.

like to thank all those involved for their support and criticism.

In preparing its final report, the Evaluation Office discussed various intermediate drafts with the GEF Secretariat and Agencies, which led to improvement in the quality of this APR. I would

Rob van den Berg Director, Evaluation Office

GEF Annual Performance Report 2007

Acknowledgments This yearâ&#x20AC;&#x2122;s annual performance report reflects the able efforts of a number of Global Environment Facility (GEF) Evaluation Office staff and consultants. Serving as task manager for general development of the report was Aaron Zazueta, Senior Evaluation Officer in the GEF Evaluation Office. Neeraj Kumar Negi, Evaluation Officer, drafted the overall document. David Michael Todd, Senior Evaluation Officer, and Anna Viggh, Evaluation Officer, conducted the capacity-building evaluation and drafted chapter 5. Ines Angulo, consultant, helped conduct the desk reviews for the capacity development evaluation and drafted chapter 9, which presents findings on management action records. Cynthia Cummis, consultant, conducted the pilot assessment of greenhouse gas

emission policieis and guidelines of GEF Agencies and drafted the related chapter 8. Appreciation is also due to colleagues in the Evaluation Office who reviewed the terminal evaluation reports under the coordination of Neeraj Kumar Negi. They include Evaluation Officers Sandra Ramboli, Lee Risby, Neeraj Kumar Negi, and Anna Viggh; Evaluation Specialist Divya Nair; and consultants Ines Angulo, Alejandro Imbach, Maria Soledad Mackinnon, and Timothy Ranja. The Evaluation Office also wishes to thank the staff of the GEF Agencies for their valuable comments and contributions to this evaluation. Nevertheless, final responsibility for the report rests firmly with the Office.

Abbreviations

xii

APR

annual performance report

FSP

full-size project

fiscal year

GEF

Global Environment Facility

GHG

greenhouse gas

Implementing Agency

IEG

Independent Evaluation Group

M&E

monitoring and evaluation

MAR MSP OPS4 PIF SGP UN UNDP UNEP

management action record medium-size project Fourth Overall Performance Study project identification form Small Grants Programme United Nations United Nations Development Programme United Nations Environment Programme

1. Background, Main Conclusions, and Recommendations 1.1 Background This document is the fourth annual performance report (APR) that the Global Environment Facil ity (GEF) Evaluation Office has developed. The report presents a detailed account of some aspects of project results, of processes that may affect these results, and of monitoring and evaluation (M&E) arrangements in completed GEF projects. In the APR for fiscal year (FY) 2005,1 the Evaluation Office presented an assessment of the M&E arrangements across the GEF partnership. The APR for FY 2006 contained an assessment of project supervision arrangements across the GEF partnership. In this APR, the Evaluation Office presents an assessment of the extent to which capacity development activities in GEF projects are relevant, effective, and efficient and the results, and the sustainability of the results, of these activities. It also presents a review of Agency carbon footprint policies and guidelines. For the first time, a performance matrix is presented, which summarizes the performance of the GEF Agencies and the GEF Secretariat on various parameters tracked by the Office. This matrix will be a regular feature of future APRs. This year, the Evaluation Office also piloted an approach to direct verification of terminal evaluations. The 2007 APR continues the annual presentation of assessment of project outcomes, project sustainability, project completion delays, materialization

of cofinancing, and quality of monitoring. For the assessment of project outcomes, project sustainability, and delays in project completion, 148 projects for which terminal evaluation reports were submitted by the GEF Agencies to the Evaluation Office in FY 2005 (41 projects), FY 2006 (66 projects), and FY 2007 (41 projects) were considered. The GEF investment in these 148 projects totaled $713 million. The focus of this APR’s reporting is on the 41 projects for which terminal evaluation reports were submitted in FY 2007; these represent a total GEF investment of $199 million. For assessment of materialization of cofinancing, all 223 projects for which terminal evaluation reports have been submitted since January 2001 were considered. Of these, information on actual materialization of cofinancing was provided for 157 projects (70 percent), either in the terminal evaluation reports and/or by the relevant Implementing Agency (IA). The GEF has invested a total of $782 million in these 157 projects; the IAs reported that an additional $2.5 billion was leveraged for these projects in the form of cofinancing. To assess capacity development activities in GEF projects, the Evaluation Office conducted detailed country case studies in Vietnam and the Philippines. In addition, 42 terminal evaluations, including 41 submitted to the Office during 1

FY 2007, were reviewed. The Office was thus able to assess the extent to which findings based on the analysis of the data from these two approaches were consistent. The GEF Evaluation Office commissioned a survey of the policieis and guidelines of GEF Implementing and Executing Agencies to broadly assess the extent to which they are addressing the greenhouse gas (GHG) emissions generated by their internal operations. GHG footprint–related policieis and guidelines of the 10 GEF Agencies were reviewed. This year’s management action record (MAR) tracks the level of adoption of 46 Council decisions based on 12 GEF Evaluation Office documents. The Office was able to verify 41 of these decisions and plans to carry out thematic assessments in the future to assess adoption of Council decisions that were not verified this year. This APR marks the first time a performance matrix providing a summary of the performance of GEF Agencies and the GEF Secretariat on relevant parameters has been included. Several of the parameters included in the matrix are already assessed on an annual basis by the Evaluation Office. For the remainder, the Office—in collaboration with the evaluation units of the GEF Agencies—is developing assessment approaches. Reporting on such parameters will be provided in future APRs. The APRs for 2005 and 2006 found that the absence of direct verification of completed projects prevents a more comprehensive assessment of the veracity of the submitted terminal evaluation reports. The APR evaluation primarily involves review of the evidence presented in the terminal evaluation reports, with verification of performance ratings based primarily on desk reviews. Although this methodology enables the

Office to make the performance ratings more consistent with the evidence provided in the terminal evaluation reports, it does not allow it to establish the veracity of the evidence presented. Direct verification permits the Office to address this concern. In addition, the field verification process provides the Office with an opportunity to gather new information to draw lessons from the completed projects. To develop its approach to direct verification further before adopting it as a regular activity of the APR, the Evaluation Office this year piloted the approach for three full-size projects (FSPs) for which terminal evaluation reports had been submitted in FY 2007. One of the limitations noted in the earlier APRs has been that, on many issues, the number of projects for which data were available was not sufficient to allow for in-depth assessment of differences in performance and the underlying factors that affect performance. With the inclusion of the FY 2007 cohort in the data pool, this constraint has been mitigated to a large extent for such issues as trends pertaining to the quality of terminal evaluation reports and reported materialization of cofinancing. Analysis of other issues remains constrained, but additional years of data will similarly answer to these.

1.2 Findings and Conclusions Results Conclusion 1: The percentage of completed projects with outcome ratings in the satisfactory range is close to the 75 percent target specified in the GEF-4 replenishment agreement. Among the completed GEF projects assessed and rated this year, 73 percent were rated moderately satisfactory or above in achievement of outcomes, and 59 percent were rated moderately likely or above in sustainability of outcomes. Although the

GEF Annual Performance Report 2007

outcomes of 60 percent of FSPs and 85 percent of MSPs were rated moderately satisfactory or above. This is unlike the FY 2006 cohort, where the outcome ratings for FSPs and MSPs were similar.

outcome ratings for the FY 2007 cohort are lower than those for FYs 2005 and 2006, no conclusions can yet be drawn as to whether this will constitute a trend. Attainment of project outcomes. The Evalua tion Office rated the achievement of project out comes on criteria of relevance, effectiveness, and efficiency. Of the 41 terminal evaluation reports submitted in FY 2007, 40 reports (98 percent) provided sufficient information to allow assessment of the level of attainment of project outcomes (see table 1.1).2 The key findings of this assessment are as follows. zz

Of the 40 projects whose outcomes were rated by the Evaluation Office, 73 percent were rated moderately satisfactory or above. Although this figure is lower than that for FY 2005 (82 percent) and FY 2006 (84 percent), given the relatively small number of observations and the differences in project mix, the difference is not statistically significant.

Of the total investment in the rated projects ($198 million), 69 percent ($137 million) was allocated to projects that were rated moder ately satisfactory or above.

Project performance for the FY 2007 cohort is close to the target set for the fourth GEF replenishment of 75 percent of projects having satisfactory outcomes (GEF 2006).3 Although the completed projects assessed for FY 2007 do not fall under the provisions of the fourth replenishment, their performance indicates that the target outcome achievement is realistic. Sustainability of project outcomes. The Evaluation Office rated sustainability based on its assessment of level of risks to sustainability of outcomes on four dimensions: financial, sociopolitical, institutional and governance, and environmental. Of the terminal evaluation reports submitted in FY 2007, 39 (95 percent) provided sufficient information to allow assessment of sustainability of project outcomes. The key findings of this assessment follow: zz

There were differences in the outcome ratings of FSPs and medium-size projects (MSPs): the

Of the 39 projects rated, the sustainability of outcomes of 59 percent (23 projects) was deemed moderately likely or above. Although outcomes of 65 percent of projects from the

Table 1.1

Summary of Project Outcomes and Sustainability Ratings Factor

FY 2005

FY 2006

FY 2007

Number of terminal evaluation reports submitted

Number of projects with outcome rating

% rated moderately satisfactory or above in outcome ratings

Number of projects with sustainability of outcomes ratings

% rated moderately likely or above in sustainability of outcomes ratings

Number of projects rated on both outcomes and sustainability of outcomes

% of rated projects with moderately satisfactory/moderately likely or above for both

Note: The methodology used to rate project outcomes for FY 2007 was the same as that used for FY 2006.

1. Background, Main Conclusions, and Recommendations

FY 2006 cohort had been similarly rated, the difference between the two cohorts is not statistically significant. zz

Of the total GEF investment in rated projects ($182 million), 55 percent ($99 million) was invested in projects that were rated moderately likely or above in terms of the sustainability of their outcomes.

The Evaluation Office assessed the extent to which projects that were rated moderately satisfactory or above in achievement of outcomes were also rated moderately likely or above in sustainability of outcomes. Of the terminal evaluation reports in the FY 2007 cohort, 39 (95 percent) provided sufficient information on both the parameters to permit assessment. The Office found that of the rated projects from the FY 2007 cohort, 51 percent were rated both moderately satisfactory or above in outcomes and moderately likely or above in sustainability. In terms of GEF investment, 44 percent was invested in these projects. The corresponding figures for the FY 2006 and FY 2005 cohorts were 61 and 56 percent, respectively.

Processes Conclusion 2: The materialization of cofinancing reported by the IAs was about three-fourths of that promised at project approval. There are great variations among projects in terms of level of GEF investment, promised cofinancing, and reported materialized cofinancing. The figures for a cohort could easily be skewed by a few projects. Consequently, the average figures for cohorts may fluctuate despite the absence of an underlying trend. The cofinancing figures reported by the IAs for the FY 2007 cohort need to be noted with this caveat.

Of the 41 terminal evaluation reports submitted for FY 2007, 33 reported on materialization of cofinancing. Of the eight projects for which terminal evaluation reports did not provide information on cofinancing materialization, this information was retrieved for six by the Evaluation Office through follow-up with the respective IA. For the FY 2007 cohort, an average of $2.60 had been promised at the point of project inception in cofinancing per dollar of approved GEF investment. Of this, $1.90 was reported to have materialized. For FY 2006, in comparison, $2.10 had been promised and $2.40 was reported to have materialized (see table 1.2). If all terminal evaluation reports submitted to the Evaluation Office to date are taken into con sideration, information on cofinancing is available for 157 projects. For these, the Agencies promised an average of $3.50 in cofinancing per $1.00 of approved GEF grant. The actual cofinancing reported was slightly lower: $3.20 per $1.00 of approved GEF grant. Thus, an average of 92 percent of promised cofinancing was reported to have materialized. On average, the projects of the FY 2007 cohort were completed after a delay of 13 months; 14 percent were completed after a delay of at least two years. The Evaluation Office began tracking project completion delays in FY 2005. The average project completion delay was 19 months for the FY 2005 cohort and 13 months for the FYs 2006 and 2007 cohorts. Compared to 44 percent of the projects in the FY 2005 cohort, only 17 percent of the projects in the FY 2006 cohort and 14 percent of the FY 2007 cohort had delays of two years or more. It is too early to determine if this drop in average delay in project completion constitutes a trend.

GEF Annual Performance Report 2007

Table 1.2

Materialization of Cofinancing FY of report submission 2004

2005

2006

2007

Up to All 2006 years

118

157

118

151

6.2

5.5

5.9

6.4

3.5

4.8

5.0

6.1

4.6

6.3

3.3

4.4

4.6

4.5

29.5

8.4

43.2

9.5

7.2

12.8

19.0

17.4

1.50

2.10

2.60

8.2

9.2

2.40

1.90

Factor

2002

Number of projects with cofinancing data available

Data obtained from terminal evaluation reports

Data otherwise obtained Approved GEF grant per project (million $) Actual GEF grant per project (million $) Promised cofinancing per project (million $) Promised cofinancing per $1.00 of approved GEF grant Reported materialized cofinancing per project (million $) Reported materialized cofinancing per $1.00 of approved GEF grant Materialized cofinancing per $1.00 of promised cofinancing (%)

Conclusion 3: The results of capacity development activities in completed GEF projects are generally positive and in some areas significant. However, in many instances, the gains made are not sustained. In FY 2007, the Evaluation Office began work to assess GEF capacity development activities. To date, the evaluation team has completed literature reviews and country case studies for the Philippines and Vietnam. The case studies showed that the results of capacity development activities are generally positive and, in some areas, significant. GEF capacity development support has been relevant and in line with national policy priorities; it has also reflected a significant amount of national ownership. The effectiveness of capacity development activities has varied, although even in areas that did not produce immediate benefits, such may develop in the longer term. The cost-effectiveness of capacity development activities was difficult to assess. Although there are many improvements to capacity at the individual, institutional, and systemic levels, there are doubts about sustainability. First, staff turnover in many government institutions is high, so there is a need to repeat training

1. Background, Main Conclusions, and Recommendations

4.70 29.2 4.70 99

2003

1.50 7.1 1.30 85

7.40 38.9 6.60 90

10.0 1.60 106

114

3.80 18.3 3.60 96

3.5 16.0 3.2 92

regularly. Second, once staff have been trained and put their new skills into practice, they often need higher level skills which require additional learning opportunities. The case studies revealed a common underlying weakness in the training programs, namely the tendency to plan and execute training as a “one-shot” solution. Training approaches should be built on the basis of existing capacity in a country or region, such as universities or specialist public or private sector training institutions. Forty-two terminal evaluations, including 41 submitted to the Evaluation Office in FY 2007, were reviewed for this assessment. The findings of the terminal evaluation review supported those of the country case studies. The results of capacity development activities have generally been positive. Support has overall been relevant to national development goals, with the exception of some specific training exercises in five projects. The efficiency of capacity development activities has usually been satisfactory in meeting output and outcome targets; some projects, however, have suffered implementation delays. The cost-effectiveness of

capacity development activities was difficult to assess, mainly because budget information was reported for such only when it was a separate project component. The effectiveness of such activities has varied, and there are doubts concerning the sustainability of results. The sustainability of capacity improvements from training remains low because few national or regional long-term training programs were developed. The review of terminal evaluations found that there is inadequate reporting on the performance of capacity development activities implemented by projects. The weak evidence base of capacity development activities makes it difficult to learn lessons or track accountability of project investments. The country case studies and the review of terminal evaluations provide the following lessons on capacity development activities. Capacity development at the systemic level must be realistic. Overly ambitious goals to change policies or laws may, when not attained, adversely affect implementation of other project components. Institutional strengthening requires baselines and assessments to determine how new capacity will be absorbed. Long-term training programs and capacities should be based on a national or regional context rather than a dependence on one-shot training.

Monitoring and Evaluation Conclusion 4: There is a strong association between quality at entry of M&E arrangements and actual quality of monitoring during implementation. Of the 41 terminal evaluation reports submitted during FY 2007, 33 (80 percent) provided sufficient information on M&E to allow the Evaluation Office to rate quality of project monitoring during implementation. Of the rated projects, 61 percent were rated moderately satisfactory or above on quality of monitoring (see table 1.3).

Table 1.3

M&E Ratings for Projects FY of terminal evaluation report submission Factor

2004 2005 2006 2007

No. of terminal evaluation reports submitted

Did not report on M&E

Did report on M&E

Reporting not required % rated moderately satisfactory or above

Of the 41 projects, quality of M&E arrangements at entry was assessed for 40. Of these, 27 (68 percent) were rated as moderately satisfactory or above. The Evaluation Office continued with the analysis presented in the FY 2006 APR where the association between the ratings for quality of M&E arrangements at entry and quality of project monitoring during implementation was assessed. Such an assessment was possible for only 32 projects. Of the 24 projects in the 2007 cohort whose quality of M&E arrangements at entry was rated to be moderately satisfactory or above, for 18 (75 percent), the quality of project monitoring during implementation was rated to be in the same range. In contrast, of the eight projects whose quality of M&E arrangements at entry was rated to be moderately unsatisfactory or below, only one (13 percent) was rated as moderately satisfactory or above in quality of project monitoring during implementation. This confirms the findings of the FY 2006 APR, where a strong association between the two had been reported. Conclusion 5: There has been significant improvement in the overall quality of terminal evaluation reports. However, further improvements are required in the reporting of financial information.

GEF Annual Performance Report 2007

In FY 2007, 41 terminal evaluation reports were submitted. This is considerably lower than the 66 submitted during FY 2006. The Evaluation Office will investigate this phenomenon and report on it in the next APR. Of the 41 terminal evaluation reports submit ted in FY 2007, the quality of 39 (95 percent) was rated to be moderately satisfactory or above. This performance is better than that of earlier cohorts (see table 1.4). On all performance parameters assessed, a greater percentage of the terminal evaluation reports provided sufficient information, thus enabling the Evaluation Office to assess the performance of a greater percentage of projects. The underlying trend of improvement in the quality of terminal evaluation reports submitted to the Evaluation Office becomes stronger when the year of terminal evaluation report completion, instead of year of submission, is used as a basis for comparison. This improvement is a result of the quality control measures adopted by the GEF Agencies and guidance provided by the GEF Evaluation Office. Table 1.4

Quality of Terminal Evaluation Reports FY of report submission Factor

2004

2005

2006

2007

No. of reports submitted

% without sufficient information on project outcomes

–

% without sufficient information on sustainability of outcomes

–

% that did not report on cofinance

% without sufficient information on M&E

% rated moderately satisfactory or above in quality

1. Background, Main Conclusions, and Recommendations

Despite improvement in the overall quality of reporting, financial reporting and analysis presented in the terminal evaluation reports remains an area where there has been little improvement. Almost half of the terminal evaluation reports (44 percent) did not provide adequate information on aspects related to financial performance. Consequently, it is difficult to assess the variance in actual financial performance and the effect of such variation on results achieved by these projects. Financial reporting is primarily contingent on the extent to which the respective Agency provides timely financial information for terminal evaluation. This again is dependent on the financial systems of the respective Agency. For example, of the reports submitted to the Office since FY 2005, 70 percent of those from the World Bank provide adequate financial information, compared to only 41 percent from the United Nations Environment Programme (UNEP) and 37 percent from the United Nations Development Programme (UNDP). This finding is not surprising, in that the Bank is a financial institution that accords greater attention to financial reporting. Since it is difficult to make changes in financial systems, this remains an area where more effort is required. The Evaluation Office will devote more attention to financial reporting and will work with the GEF Agencies to find ways to improve performance on this parameter. In November 2007, after the “Joint Evaluation of the Small Grants Programme” was presented to the GEF Council, information emerged regarding ongoing investigations of which the evaluation team was not aware. These investigations were confidential. The Evaluation Office will fine-tune the guidelines on how confidential information on GEF activities should be shared with evaluators while maintaining the confidentiality of the Agencies.

Greenhouse Gas Footprint Conclusion 6: The GEF Agencies are on track in addressing the greenhouse gas emissions of their internal operations. However, most are in early stages of developing and adopting a comprehensive greenhouse gas management strategy. The GEF Evaluation Office commissioned a survey of the policieis and guidelines of GEF Agencies to broadly assess the extent to which they are addressing the GHG emissions generated by their internal operations. GHG footprint–related policieis and guidelines of the 10 GEF Agencies were reviewed. This assessment was primarily based on a review of publicly available information. Supplementary information was gathered for some Agencies by requesting relevant documents and through phone conversations. Since several of the Agencies are in the process of defining their approach to addressing GHG emissions from their operations, some of the information in this report might be outdated and contain gaps. A set of climate evaluation criteria was developed for a review of the GHG reduction approaches of the 10 Agencies. The climate policieis and guidelines of the Agencies were compared to these criteria. The findings show that the GEF Agencies are on track in addressing the GHG emissions of their internal operations, but that most are in an early stage. Some Agencies do have commitments to further address their internal impacts in the future. For example, Agencies under the United Nations (UN) umbrella will be working toward reducing their GHG emissions as part of a UNwide climate-neutral target that they plan to meet in the future. Among the Agencies reviewed, the European Bank for Reconstruction and Development, the Inter-American Development Bank, and UNEP have begun to address most of the 8

criteria considered for assessment. Overall, the banks have made more progress in addressing the climate impacts of their internal operations than have the other Agencies. All Agencies need to advance their efforts to address their internal climate change impacts and further develop processes and systems to track and manage their internal GHG inventory data. The Agencies should first focus their efforts on completing a robust GHG inventory. Once they have assessed their climate risk through such an inventory, they will be well positioned to develop a cost-effective and disciplined approach to reducing their impacts.

Management Action Record Conclusion 7: All 41 verified Council decisions in the 2007 Management Action Record show a level of adoption of medium and higher. For the older Council decisions, 14 of 27 show substantial progress in level of adoption as compared to last year. This year’s MAR tracks the level of adoption of 46 Council decisions based on 12 GEF Evaluation Office documents by presenting ratings from GEF management and verification of these ratings by the Evaluation Office. The Office was able to verify the adoption of 41 of these 46 decisions (see table 1.5). Decisions that could not be verified yet are mainly related to project M&E standards and quality of supervision. The Evaluation Office will carry out thematic assessments in the future to assess adoption of these Council decisions. The Office rated 18 (44 percent) of the 41 verified decisions as having been adopted by management at high or substantial levels. This represents an improvement over last year, when 33 percent were in this range. In addition, 12 Council decisions thus far have been fully adopted and therefore have been graduated from the MAR. Six of GEF Annual Performance Report 2007

Table 9.1

Ratings of GEF Progress in Adopting Council Decisions GEF Evaluation Office rating Management rating

High

Substantial

Medium

Negligible

Not possible to verify yet

Sum of management ratings

High

Substantial

Medium

Negligible

Not possible to verify yet

Sum of Office ratings

Note: Highlighted fields show agreement between the ratings of management and the GEF Evaluation Office; fields to the right of the diagonal represent higher ratings by management than by the Evaluation Office (except in the case of not possible to verify yet).

these were adopted during FY 2006 and pertained to the Evaluation of the Experience of Executing Agencies under Expanded Opportunities in the GEF. The remaining six were based on other evaluations and were fully adopted during FY 2007. Table 1.5 shows that the Evaluation Office and GEF management agreed on the rating of progress of adoption for 51 percent of Council decisions (21 of 41). For 46 percent of the decisions (19 of 41), the Evaluation Office gave a lower rating than that given by management. Many of these lower ratings reflect the fact that even though the GEF has taken measures to address the decisions, these cannot yet be considered to demonstrate a high level of achievement. They also reflect the fact that some proposals to the GEF Council have not yet been approved. If and when the Council does approve these proposals, substantial adoption is likely to occur. Thirty Council decisions from seven GEF Evaluation Office documents were included in both last year’s and this year’s MAR. The Evaluation Office was able to follow the progress of adoption of 27 of these Council decisions (3 were rated as not possible to verify yet). Of these 27 Council decisions, 52 percent (14 out of 27) showed progress in their level of adoption. 1. Background, Main Conclusions, and Recommendations

The Evaluation Office found noticeable progress over last year in adoption of Council decisions requesting increased transparency in the GEF project approval process through an improved management information system. The Office assessed the recent launch of a new system, which includes a country portal to provide information related to GEF projects at the country level, to be a concrete improvement. Eight out of the nine Council decisions for which ratings have not shown improvement since last year are related to the local benefits study and the biosafety evaluation. Regarding the former, Council decisions stress the need to establish a system that ensures that local benefits are addressed in a more systematic way at all stages of the GEF project cycle. The GEF Secretariat currently cannot verify the quality of this aspect in project design or implementation because it still has no system in place to involve specialist social development expertise in its project review processes. Regarding biosafety, the Evaluation Office will rate substantial adoption once the Council approves the “Program Document for GEF Support to Biosafety during GEF-4.” The Small Grants Programme (SGP) management has started to address various Council decisions 9

from the SGP joint evaluation through the GEF SGP Steering Committee. However, because the country coordinators are not represented on the Steering Committee, there is a risk that, inadvertently, new proposals will not consider country operational issues and perspectives sufficiently. The inclusion of some senior country coordinators on the Steering Committee would allow for the country program perspective to be heard in the discussions that lead to SGP proposals addressing GEF Council decisions. A complete version of the MAR is available at the GEF Evaluation Office Web site (www.gefeo.org).

Performance Matrix The performance matrix provides a summary of the performance of the GEF Agencies and GEF Secretariat on 13 parameters, covering key areas such as results, processes affecting results, efficiency, M&E, and learning. Several of the parameters included in the matrix are already assessed by the Evaluation Office on an annual basis. Since performance ratings on these parameters fluctuate from year to year, running averages of two to four years, depending on the parameter, will be used in the matrix (see chapter 9 and annex D for methodological details on the performance matrix). Of the 13 parameters included in the performance matrix, ratings have been provided on 9. Note, however, that the information provided for parameter 13, improvement in performance, addresses only 1 of the parameter’s 12 dimensions. In the future, as data for more years become available, it will be possible to track improvement in performance on a greater number of dimensions. Based on the review of terminal evaluation reports submitted to the Evaluation Office since FY 2005, the Office rated outcome achievement (parameter 1) in 81 percent of the projects to be moderately satisfactory or above. As noted, this percentage is 10

higher than the 75 percent target specified in the GEF-4 replenishment agreement, even though these projects are not subject to this provision. The percentage of World Bank projects (87 percent) with ratings in the satisfactory range has been significantly higher than the target. For parameter 13, improvement in performance, the quality of terminal evaluations has been assessed. As noted in conclusion 5 above, there has been substantial improvement in performance on this dimension. UNEP and UNDP demonstrated considerable improvements, and the quality of a high percentage of terminal evaluation reports submitted by the World Bank continues to be in the satisfactory range.

1.3 Recommendation Recommendation 1: The GEF Secretariat, in coordination with the GEF Agencies, should conduct a formal and in-depth survey to more accurately and thoroughly assess GEF Agency efforts to reduce GHG emissions.

1.4 Issues for the Future zz

The GEF Evaluation Office will investigate why the number of terminal evaluations submitted during FY 2007 was lower than expected and will report on this in the next APR.

The GEF Evaluation Office will work with the GEF Secretariat to develop better systems to track project completion.

The GEF Evaluation Office will fine-tune the guidelines on how confidential information on GEF activities should be shared with the evaluators while maintaining the confidentiality of the Agencies.

The GEF Evaluation Office will work with the GEF Secretariat, GEF Trustee, and GEF Agencies to facilitate better financial reporting for completed projects. GEF Annual Performance Report 2007

Notes 1. The GEF fiscal year, like that of the World Bank, runs from July 1 to June 30. FY 2007, the primary focus of this report, comprises the period from July 1, 2006, to June 30, 2007.

1. Background, Main Conclusions, and Recommendations

2. The complete list of the terminal evaluation reports reviewed for FY 2007 is provided in annex B. 3. The GEF-4 replenishment period runs from 2006 to 2010.

2. Scope and Methodology 2.1 Scope Each year, the APR provides an assessment of the performance of completed GEF projects, analysis of the processes that affect accomplishment of results, and findings of the GEF Evaluation Office’s oversight of project monitoring and evaluation activities across the portfolio. The APR also provides the GEF Council, other GEF entities, and stakeholders with feedback to help improve the performance of GEF projects. Some issues are addressed by the APR annually, some biennially; others are addressed whenever there is a need to do so. The 2007 APR includes the following: zz

An overview of the extent to which GEF projects are achieving their objectives (chapter 3). This overview consists of the Evaluation Office’s assessment of the extent to which the completed projects for which terminal evaluation reports were submitted in FY 2007 achieved expected outcomes and the risks to sustainability of achieved outcomes. The APR will continue to report annually on attainment of objectives and outcomes, and on sustainability of outcomes. A presentation of the materialization of project cofinancing and delays in project completion reported by the GEF Agencies (chapter 4). This APR reports on the extent

to which cofinancing promised at the point of project endorsement has materialized and on delays in project completion. The assessment on materialization of cofinancing is based on figures reported by the respective GEF Agencies. The Evaluation Office will continue to report on these issues on an annual basis. zz

A detailed assessment of the extent to which capacity development activities in GEF projects are relevant, efficient, and effective and on the results and sustainability of these activities (chapter 5). This is the first time the Evaluation Office has undertaken this assessment. The experience gained through recent field studies undertaken by the Office in the Philippines and Vietnam facilitated the development of an effective methodology for reviewing GEF project capacity development targets and achievements as reported by the terminal evaluation reports submitted to the Office. For this assessment, 42 terminal evaluations, including 41 submitted to the Office during FY 2007, were reviewed.

An assessment of the quality of project monitoring (chapter 6). This includes an examination of the quality of M&E at project completion and an assessment of the quality at entry of M&E arrangements of completed projects. The APR reports on these M&E issues on an annual basis.

An assessment of the quality of terminal evaluation reports submitted by the GEF Agencies to the Evaluation Office (chapter 7). This assessment, which is presented annually, provides information on the quality of terminal evaluation reports by focal area and Agency and assesses trends in terminal evaluation report quality. This year, the Office piloted a methodology for direct verification of terminal evaluations. Findings of direct verification will be included in future APRs. A review of the policies and guidelines of the GEF Agencies in addressing the carbon footprint of their operations (chapter 8). For the first time, the Evaluation Office has reviewed the policies and guidelines of the GEF Agencies addressing the carbon footprint of their operations. The review involved a desk study of these policies and guidelines and interviews with relevant staff members of the respective Agencies. A presentation of findings on management action records (chapter 9). This assessment, which is presented annually, reviews and follows up on the implementation status of evaluation recommendations that have been accepted by management and/or the GEF Council. A presentation of the performance matrix (chapter 10). This assessment, which is being presented for the first time in this APR, will be presented annually. It summarizes the performance of the GEF Implementing Agencies and the GEF Secretariat on key performance parameters.

2.2 Methodology Ensuring Reliability and Timeliness of Terminal Evaluation Reports The project terminal evaluation reports submitted by the GEF Agencies to the Evaluation Office form the core information source for much of the APR, 2. Scope and Methodology

particularly for those topics that are reported on annually. Ensuring the reliability of these reports is therefore critical. The Evaluation Office seeks to assess and strengthen this reliability in several ways, as described below. The Evaluation Office reviews terminal evalua tion reports to determine the extent to which they address the objectives and outcomes set forth in the project document, to evaluate their internal consistency, and to verify that ratings are properly substantiated. The reports are reviewed by Evaluation Office staff using a set of detailed guidelines to ensure that uniform criteria are applied (see annex A for these guidelines). When deemed appropriate, a reviewer may propose to upgrade or downgrade project ratings in a terminal evaluation report. The reviews are also examined by a peer reviewer with substantial experience in reviewing termi nal evaluations. The peer reviewer provides feed back on the review, which is incorporated by the primary reviewer in subsequent versions of the review. When a project is downgraded below moderately satisfactory (for outcomes) or below moderately likely (for sustainability), a senior evaluation offi cer in the GEF Evaluation Office also examines the review to ensure that the new ratings are justified. The reviews are then shared with the Agencies, and, after their feedback is taken into con sideration, the reviews are finalized. If a terminal evaluation report provides insuffi cient information to make an assessment or verify the Agency ratings on outcomes, sustainability, or quality of project M&E systems, the Evaluation Office classifies the corresponding project as unable to assess and excludes it from any further analysis on the respective dimension. 13

The most pervasive limitation of this review process is that it is ultimately based on the information provided by the terminal evaluation reports. Piloting of direct verification of the reports for a randomly selected sample of completed projects was aimed at developing a methodology to address this limitation in future. This year, verifications were conducted for three completed full-size projects for which terminal evaluation reports had been submitted to the Evaluation Office during FY 2007. Beginning with next year’s APR, field verification will be incorporated in the terminal evaluation review process and these findings presented. Another way to address the reliability concerns pertaining to terminal evaluation reports is to work with the GEF Agencies to more fully engage their evaluation units in the process and, when necessary, to strengthen their independence. Presently, the World Bank’s terminal evaluation process meets most of the concerns of the GEF Evaluation Office. The Bank’s Independent Evaluation Group (IEG) conducts desk reviews of all the implementation completion reports produced by management for FSPs and conducts direct verification of these reports for a sample of projects. Beginning in FY 2006, the UNEP Evaluation Office provided ratings and commentary on the quality of the terminal evaluation reports for its completed GEF projects. This fiscal year, it increased the scope of its commentaries by also assessing project outcomes, sustainability of outcomes, and implementation of M&E based on the evidence provided in the terminal evaluation reports. However, since only a few assessments have been completed so far, it is still too early to determine the overall reliability of the UNEP ratings. Similarly, the UNDP Evaluation Office this fiscal year began to provide commentary on the quality of terminal evaluations for some of its completed GEF

projects. The UNDP Evaluation Office has yet to begin reviewing project performance in terms of outcomes, sustainability of outcomes, and implementation of M&E. The GEF Evaluation Office will continue its dialogue with the GEF Agencies, while reviewing their terminal evaluation reports and verifying their ratings. The GEF Evaluation Office has been tracking the time between project completion and submission of terminal evaluation reports and between report completion and submission. Although the time lag between completion and submission has been declining, many reports are still being submitted after more than the two months that is the negotiated norm for this task. The World Bank has a system for automatic submission of an electronic version of implementation completion reports for its GEF-supported projects when these reports are completed. The other GEF Agencies have not yet developed such systems. The Office will work with the Agencies so that timely submission of terminal evaluation reports can be ensured.

Data Limitations The Evaluation Office used the F-test and chi square test to assess differences among groups of projects, and the findings reported here are sig nificant at the 90 percent or higher confidence level. Regression analysis was used to assess the magnitude and direction of change associated with different variables. This fiscal year, there was improvement in the quality of information provided in terminal evaluation reports on parameters such as sustainability and project M&E, which in the past have been identified as areas where the terminal evaluations are deficient in providing adequate information. However, there has been little improvement in the quality of information provided on materialization of cofinancing. If sufficient information on a performance parameter for a project has not been provided in GEF Annual Performance Report 2007

its terminal evaluation report, that project has not been included in the portfolio-level assessment for that parameter. Since data are now available for a greater number of completed projects and over a longer period of years, the GEF Evaluation Office is better able to predict trends and assess differences in performance. However, assessing the performance of completed projects in terms of their outcomes, sustainability of outcomes, and implementation of M&E reflects actions that are now long past, limiting the extent to which information gathered from analysis of these data is useful in making real-time corrections in operations. Notwithstanding this limitation, this assessment provides a long-term perspective on the extent to which GEF projects are achieving their objectives.

Assessment of Capacity Development Activities In this APR, the Evaluation Office presents an assessment of the extent to which capacity development activities in GEF projects are relevant, effective, and efficient, and the results and sustainability of the results of these activities. Capacity-building activities were assessed at the individual, institutional, and systemic levels. Detailed country case studies were conducted in Vietnam and the Philippines. The 42 terminal evaluation reports, including 41 submitted to the Office during FY 2007, were also reviewed. This allowed the Office to assess the extent to which findings based on the analysis of the data from these two approaches were consistent.

Review of Carbon Footprint–Related Policies and Guidelines This APR presents a set of criteria to track progress by the GEF Agencies in addressing their carbon footprint. The criteria are based on emerging institutional approaches to reducing GHG emissions 2. Scope and Methodology

and are relevant to the GEF Agencies and entities. Using these criteria, the policies and guidelines of the GEF Agencies have been assessed.

MARs Assessment Management action records facilitate review and follow-up on the implementation status of evaluation recommendations that have been accepted by management (that is, the GEF Secretariat and/ or the GEF Agencies) and/or the GEF Council. For each MAR, the Evaluation Office completes the columns pertaining to recommendations, management responses, and Council decisions. Management is invited to provide a self-rating of the level of adoption of Council decisions on recommendations and add any comments necessary. After management’s response is included in a MAR, the Evaluation Office verifies actual adoption and provides its own ratings, with comments, in time for presentation to the Council.

Performance Matrix This APR marks the first appearance of the performance matrix, which provides a summary of the performance of the three GEF Implementing Agencies and the GEF Secretariat on relevant parameters. Performance on most of these parameters, such as project outcomes, implementation completion delays, materialization of cofinancing, quality of M&E during project implementation, and quality of project terminal evaluations, is already being assessed annually by the GEF Evaluation Office. Performance on other parameters, such as quality of supervision and adaptive management, realism of risk assessment, and quality of project M&E arrangements at entry, is being assessed every two or three years. For these latter parameters, performance will be updated every few years through special appraisals. The information presented in the performance matrix of this APR (see table 10.1) is from the APRs for 15

FYsÂ 2005 and 2006. For assessing performance on project preparation elapsed time, assessments will be presented based on the GEF Project Management and Information System database. At present, the information provided by the database on this parameter is not reliable, so such assessments will be carried out when there is a substantial improvement in the relevant information. Four parameters that have been included in the performance matrix will require the development of new methodologies and approaches. This APR reports on progress in developing such methodologies and approaches for assessing three of these parameters: independence of terminal evaluations, independence of GEF partner Agency evaluation units, and quality of learning. Assessment of robustness of program result

indicators and tracking tools will be addressed in the future. Given the highly specialized and technical nature of this assessment, the GEF Evaluation Office will undertake it as part of the program studies conducted for the Fourth Overall Performance Study (OPS4) being prepared in FYâ&#x20AC;Ż2008 and will enlist assistance from appropriate technical experts.

Review of Findings The preliminary findings of this report were presented to and discussed with the GEF Secretariat and GEF Agencies during an interagency meeting held in Washington, D.C., in March 2008. Individual reviews of project terminal evaluation reports were shared with the Agencies and GEF Secretariat for comment.

GEF Annual Performance Report 2007

3. Project Outcomes and Sustainability This chapter discusses verified ratings on outcomes and sustainability for 41 projects (20 full-size and 21 medium-size projects) for which terminal evaluation reports were submitted in FY 2007. This is the third time the Evaluation Office has rated outcomes and sustainability of GEF projects. Seventy-three percent of the projects assessed this fiscal year were rated moderately satisfactory or above with regard to attainment of project outcomes. Although the percentage of projects rated in the satisfactory range is lower than for FYs 2005 (82 percent) and 2006 (84 percent), this difference is not statistically significant (see figure 3.1). More than half (58 percent) of the projects assessed in FY 2007 were rated moderately likely or above in terms of the sustainability of their outcomes; this was true for almost two-thirds of the projects Figure 3.1

Percentage of Projects with Outcomes Rated Moderately Satisfactory or Above Percentage of projects 100 82% 80 60 40 20 0

84%

in the FY 2006 cohort (65 percent) and just under half of those in the FY 2005 cohort (49 percent) (see figure 3.2). Similarly, 51 percent of FY 2007 projects were rated both moderately satisfactory and above in their outcome ratings and moderately likely or above in their sustainability ratings; the comparable percentages from prior years were 61 percent in FY 2006 and 44 percent in FY 2005. Because of fluctuations in cohort sustainability ratings, it is too early to conclude whether the lower percentages for this year’s cohort constitute a trend. Figure 3.2

Percentage of Projects with Sustainability Rated Moderately Likely or Above Percentage of projects 100 80 60 49% 40 20 0

2005 (N = 39)

65%

2006 (N = 54)

59%

2007 (N = 39)

FY of terminal evaluation report submission 73%

3.1 Rating Approach 2005 (N = 39)

2006 (N = 64)

2007 (N = 40)

FY of terminal evaluation report submission

The Evaluation Office rated project outcomes of the FY 2007 cohort based on level of achievement of project objectives and expected outcomes on a 17

six-point scale. The criteria used to assess level of achievement included assessment of ex ante outcome relevance, actual effectiveness in achievement of outcomes, and efficiency in achievement of outcomes. Relevance and effectiveness were considered to be critical criteria; the overall rating on achievement of outcomes could not be higher than the lower rating attained on either or both of the critical criteria. The Office used the same rating approach for the FY 2006 cohort; a different rating approach had been used for FY 2005. To make the ratings comparable, the Office reassessed the 2005 cohort using the above-described approach; these ratings are referenced in the present analysis. The GEF Evaluation Office rated sustainability of outcomes based on an assessment of four key risk dimensions: financial, sociopolitical, institutional framework and governance, and environmental. Based on the evidence presented in the terminal evaluation reports, risks to sustainability of outcomes were assessed on each of these dimensions. All risk dimensions were regarded as critical; overall ratings may not be higher than the lower rating on any of these dimensions. Since the FY 2006 cohort was also assessed using this approach, its ratings are directly comparable to that for the 2007 cohort. Again, the original ratings for the FY 2005 cohort were based on a different approach. To make the ratings comparable to those for 2006 and 2007, the Office reassessed the FY 2005 cohort using the new approach for sustainability ratings; these ratings are referenced in this analysis. Continuing the analysis that was first presented in the FY 2006 APR, the Evaluation Office assessed the extent to which outcomes of projects that were rated moderately satisfactory or above in terms of their achievement of outcomes were also rated moderately likely or above in terms of the sustainability of their outcomes.

Of the 41 terminal evaluation reports submitted, 18 (44 percent) were for UNDP projects and 16 (39 percent) were for World Bank projects (see figure 3.3a). In the FY 2006 cohort, by contrast, the majority of terminal evaluation reports were for World Bank projects (64 percent), with UNDP comprising only 23 percent of the total. This shift in project mix has been taken into account in assessing the differences in ratings across fiscal years. Figure 3.3

Number of Terminal Evaluation Reports by GEF Agency and Focal Area a. By Agency

b. By focal area Multifocal Land POPs, 1 3 degrad., 1

World Bank 16

UNDP 18

Int’l waters, 2

Climate change 13

UNEP 7

Biodiversity 21

N = 41 projects

In terms of focal area representation, the FY 2007 cohort is similar to those for FYs 2005 and 2006. For all these cohorts, a majority of projects were from the biodiversity focal area (see figure 3.3b for the FY 2007 cohort). The Evaluation Office was not able to rate the level of achievement of outcomes for one project and the level of sustainability of outcomes for two projects because insufficient information was provided in the respective terminal evaluation reports. Reports that did not provide sufficient information on a particular dimension were excluded from further analysis of that dimension.

3.2 Project Outcomes Of the 40 projects in the FY 2007 cohort whose outcomes were rated by the Evaluation Office,

GEF Annual Performance Report 2007

73 percent were found to be moderately satisfactory or above (see figure 3.4a and table 3.1). Similarly, of the total investment in the rated projects ($198 million), 69 percent was allocated to projects that were rated as having moderately satisfactory or above outcomes (see figure 3.4b). Unlike FY 2006 where the outcome ratings for FSPs and MSPs were similar, for FY 2007 MSPs had higher ratings than FSPs. For example, in FY 2007, the outcomes of 85 percent of MSPs were rated as moderately satisfactory or above by the Evaluation Office compared to 60 percent for FSPs. The differences in performance by Agency and focal area were not statistically significant. By outcome criterion, all of the rated projects were found to be moderately satisfactory or above in ex ante outcome relevance, 73 percent were rated moderately satisfactory or above on effectiveness in achievement of outcomes, and 72 percent were so rated for efficiency of achievement of outcomes (see figure 3.5). The GEF investment in individual projects varied from $0.5 million to $25.0 million. Of the

Figure 3.4

Project Outcome Ratings by Number of Projects and Project Funding a. By number of projects 18%

b. By GEF investment

3% 3% 13%

5% 5% 19%

35%

14% 48%

35%

N = 40 projects; total GEF funding = $199 million Highly satisfactory

Moderately unsatisfactory

Satisfactory

Unsatisfactory

Moderately satisfactory

Highly unsatisfactory

total investment in rated projects of $198 million, $137 million (69 percent) was invested in projects whose outcomes were rated moderately satisfactory or above (see table 3.2). By Agency, the GEF investment in projects rated moderately

Table 3.1

Project Outcomes by GEF Agency and Focal Area Number of projects GEF Agency Rating

UNDP

UNEP

Focal area WB

POPs

Total

Highly satisfactory

Satisfactory

Moderately satisfactory

Subtotal Moderately unsatisfactory

Unsatisfactory

Highly unsatisfactory

Subtotal

Unable to assess

Total

Note: BD = biodiversity; CC = climate change; IW = international waters; LD = land degradation; MF = multifocal; POPs = persistent organic pollutants; WB = World Bank.

3. Project Outcomes and Sustainability

The outcomes for one project, UNDP’s Barrier Removal for the Widespread Commercialization of Energy-Efficient CFC-Free Refrigerators in China (GEF ID 445), were rated as highly satisfactory, because it had fully achieved and exceeded its expected outcomes. This project was instrumental in reducing the average energy intensity of new refrigerators sold by nearly 30 percent between project inception in 1999 and 2005 and had costeffectively contributed to reductions in GHG emissions. (Note that the outcomes of this project were rated by the Evaluation Office as moderately likely to sustain.) Outcome achievement in 14 other projects was rated as satisfactory.

Figure 3.5

Percentage of Projects with Outcome Dimensions Rated Moderately Satisfactory or Above Percentage of projects 100%

100 80

73%

72%

Effectiveness (N = 40)

Efficiency (N = 39)

60 40 20 0

Relevance (N = 41)

Among the projects that were rated as having satisfactory or above outcomes, 85 percent were rated moderately likely or above on sustainability of outcomes and 90 percent were rated moderately satisfactory or above on implementation of M&E.

satisfactory or above was 78 percent for World Bank projects, 66 percent for UNEP projects, and 60 percent for UNDP projects. By focal area, 74 percent of the GEF investment for climate change, and 62 percent for biodiversity, went toward projects whose outcomes were rated moderately satisfactory or above.

The only project the Evaluation Office rated as highly unsatisfactory in terms of its outcomes

Table 3.2

Outcome Ratings for FY 2007 by GEF Investment Million $ GEF Agency Rating

UNDP

UNEP

Focal area WB

POPs

Total

9.9

0.0

9.9

0.0

9.9

Satisfactory

20.7

0.8

14.6

5.4

13.6

16.2

0.0

0.8

0.0

36.9

Moderately satisfactory

20.8

10.1

60.0

69.4

19.9

0.6

0.0

1.0

90.9

Subtotal

51.4

10.9

74.6

74.8

43.4

16.8

0.0

0.8

1.0

136.9

Moderately unsatisfactory

17.0

0.0

10.2

27.1

0.0

27.1

Unsatisfactory

18.0

5.6

0.8

17.7

5.4

0.0

0.5

0.8

0.0

24.3

0.0

10.0

0.0

10.0

0.0

10.0

34.9

5.6

20.9

44.8

15.4

0.0

0.5

0.8

0.0

61.5

0.0

1.0

0.0

1.0

0.0

1.0

86.3

16.5

96.6

119.7

58.8

16.8

0.5

2.6

1.0

199.4

Highly satisfactory

Highly unsatisfactory Subtotal Unable to assess Total

Note: BD = biodiversity; CC = climate change; IW = international waters; LD = land degradation; MF = multifocal; POPs = persistent organic pollutants; WB = World Bank.

GEF Annual Performance Report 2007

was the World Bank–implemented Solar Development Group global project (GEF ID 595), which aimed at providing electrical services to the poor. The GEF had approved $10.0 million for this project, which became effective in FY 2001. Due to a lack of progress, the project was canceled in 2004; at that time, $1.3 million of the approved GEF grant had been spent. Formulation of this project was driven more by the priorities of the project sponsors than by realistic assessment of the market opportunities in the participating developing countries. This deficiency, coupled with an overly complicated implementation structure, led to the project performing well below expectations. The Evaluation Office rated outcome achievement for seven other projects as unsatisfactory. Among the projects that were rated unsatisfactory or below in outcomes, only 20 percent were rated as moderately likely or above to sustain the achieved outcomes, and 43 percent were rated moderately satisfactory or above on implementation of M&E.

3.3 Project Outcomes Sustainability Of the 41 terminal evaluation reports submitted in FY 2007, 2 (5 percent) did not provide sufficient information to allow assessment of sustainability of outcomes. In previous fiscal year cohorts, the corresponding percentages of reports with insufficient information for assessment of sustainability were 18 percent in FY 2006 (12 of 66 projects) and 12 percent in FY 2005 (5 of 41 projects). Thus, the extent to which terminal evaluation reports provide information on sustainability of outcomes appears to be improving substantially. Of the 39 projects that were rated on sustainability of outcomes, 23 (59 percent) were rated as moderately likely or above (see table 3.3 and figure 3.6a). In terms of GEF investment, 55 percent was invested in projects that were rated as moderately likely or above in the sustainability of their outcomes (see table 3.4 and figure 3.6b). The sustainability ratings of MSPs were higher than those for FSPs: 75 percent of the former were rated moderately likely or above on their sustainability

Table 3.3

Sustainability of Project Outcomes by GEF Agency and Focal Area Number of projects GEF Agency

Focal area

Rating

UNDP

UNEP

Likely

Moderately likely

POPs

Total

Subtotal

Moderately unlikely

Unlikely

Subtotal

Unable to assess

Total

Note: BD = biodiversity; CC = climate change; IW = international waters; LD = land degradation; MF = multifocal; POPs = persistent organic pollutants; WB = World Bank.

3. Project Outcomes and Sustainability

rated projects faced at least significant financial risks. Twenty-eight percent of the rated projects faced significant institutional and governancerelated risks (see figure 3.7).

Figure 3.6

Sustainability Ratings by Number of Projects and Project Funding a. By number of projects 10%

b. By GEF investment 13% 1%

Figure 3.7

Perceived Risks Underlying Projects Receiving Low Sustainability Ratings 53%

51% 31%

Percentage of projects 50

33%

Moderately unlikely

40 N = 39 projects; total GEF funding = $183 million Likely

Moderately unlikely

Moderately likely

Unlikely

30%

23% 20%

10%

11%

of outcomes compared to only 42 percent of the latter. This finding differs from that for FY 2006, when ratings for FSP and MSP sustainability of outcomes were similar. The reasons for this variation are not clear.

Financial (N = 37)

Sociopolitical (N = 40)

Institutional (N = 39)

Environmental (N = 30)

Sustainability of outcomes was rated unlikely for four (11 percent) projects because of high financial risks. The causes of the high level of financial risks for these projects were as follows:

Among the risk dimensions that were assessed to rate sustainability of outcomes, financial risks were the most pervasive—about 41 percent of the Table 3.4

Sustainability of Outcome Ratings for FY 2007 by GEF Investment Million $ GEF Agency

Focal area

Rating

UNDP

UNEP

POPs

Total

Likely

0.7

0.0

1.8

0.0

1.7

0.0

0.8

0.0

2.5

Moderately likely

36.3

7.8

52.8

75.1

21.2

0.6

0.0

96.9

Subtotal

37.0

7.8

54.7

75.1

22.9

0.6

0.0

0.8

0.0

99.4

Moderately unlikely

29.8

8.7

20.8

31.9

25.1

0.0

0.5

0.8

1.0

59.2

Unlikely

3.3

0.0

20.2

12.7

10.8

0.0

23.5

Subtotal

33.1

8.7

41.0

44.6

35.9

0.0

0.5

0.8

1.0

82.8

Unable to assess

16.2

0.0

1.0

0.0

16.2

0.0

1.0

0.0

17.2

Total

86.3

16.5

96.7

119.7

58.8

16.8

0.5

2.6

1.0

199.4

Note: BD = biodiversity; CC = climate change; IW = international waters; LD = land degradation; MF = multifocal; POPs = persistent organic pollutants; WB = World Bank.

GEF Annual Performance Report 2007

Lower outcome achievement by the project dissuaded potential donors from investing in the planned follow-up activities (UNDP’s Pilot Production and Commercial Dissemination of Solar Cooker project in South Africa, GEF ID 1311).

The sources of income for sustaining the follow-up activities were not as significant as had been expected (UNDP’s Conserving Mountain Biodiversity in Southern Lesotho project, GEF ID 245).

The market for the products promoted by the project were not as significant as had been expected (World Bank’s Solar Development Group project).

The financial support received from the government for continuation of project followup activities was lower than expected (World Bank’s Central Asia Transboundary Biodiversity Project, GEF ID 110).

3.4 Outcomes and Outcomes Sustainability In the FY 2006 APR, the Evaluation Office for the first time presented an assessment of the extent to which projects that are rated moderately satisfactory or above in achievement of their outcomes are also rated moderately likely or above in the sustainability of their outcomes. For FY 2007, 39 reports (95 percent) provided sufficient information to allow assessment of both project outcomes and sustainability of project outcomes. For the rated

3. Project Outcomes and Sustainability

projects of the FY 2007 cohort, 51 percent of the projects—and 44 percent of the GEF project investment—were rated moderately satisfactory or above in outcomes and moderately likely or above in sustainability (see figure 3.8). Compared to this, 61 percent of the projects and 56 percent of the investment for the FY 2006 cohort had been rated moderately satisfactory or above in outcomes and moderately likely and above in sustainability of outcomes. For the FY 2005 cohort, 44 percent of projects and 60 percent of investments had been so rated. The difference between the ratings by cohort is not statistically significant. Figure 3.8

Projects with Outcomes/Sustainability Rated as Moderately Satisfactory/Moderately Likely or Above a. By number of projects Percentage of projects 100 80 73% 60

59%

51%

40 20 0

b. By GEF investment 100 80 60 40 20 0

69%

Outcomes

(N = 40; $193.8 million)

55%

Sustainability (N = 39; $182.1 million)

44%

Outcomes and sustainability (N = 39; $182.1 million)

4. Factors Affecting Attainment of Project Results 4.1 Materialization of Cofinancing

6 of these (15 percent), this amount was less than half of that promised. The figures for materialization of cofinancing are skewed due to the inclusion of the World Bank’s Energy Efficiency Project in Brazil (GEF ID 128) in the cohort. This project alone accounts for about half of the total shortfall in cofinancing materialization. The GEF had approved a grant of $15.0 million for this project, of which $11.9 million was actually used. Of the $105.5 million promised in cofinancing for this project, only $38.1 million materialized.

Of the 41 projects for which terminal evaluation reports were submitted in FY 2007, information on materialization of cofinancing was provided for 33. The Evaluation Office followed up with the respective GEF Agencies and thereby obtained information on an additional six projects. For the 39 projects for which information on cofinancing was reported, an average of $2.60 was promised per $1.00 of approved GEF grant, and an average of $1.90 (75 percent) was reported to have materialized (see table 4.1 and figure 4.1). For 21 projects (54 percent), the materialized cofinancing was lower than the promised amount; for

In FY 2006, a lower amount of cofinancing ($2.10) had been promised per GEF dollar than in FY 2007 but a higher amount was reported to have materialized ($2.40).

Table 4.1

Materialization of Cofinancing FY of report submission 2005

2006

2007

Up to All 2006 years

118

157

118

151

6.2

5.5

5.9

6.4

3.5

4.8

5.0

6.1

4.6

6.3

3.3

4.4

4.6

4.5

29.5

8.4

43.2

9.5

7.2

12.8

19.0

17.4

1.50

2.10

2.60

8.2

9.2

2.40

1.90

2002

Number of projects with cofinancing data available

Data obtained from terminal evaluation reports Data otherwise obtained Approved GEF grant per project (million $) Actual GEF grant per project (million $) Promised cofinancing per project (million $) Promised cofinancing per $1.00 of approved GEF grant Reported materialized cofinancing per project (million $) Reported materialized cofinancing per $1.00 of approved GEF grant Materialized cofinancing per $1.00 of promised cofinancing (%)

2004

Factor

4.70 29.2 4.70 99

2003

1.50 7.1 1.30 85

7.40 38.9 6.60 90

10.0 1.60 106

114

3.80 18.3 3.60 96

3.5 16.0 3.2 92

Figure 4.1

Reported Materialization of Cofinancing per Dollar of Approved GEF Grant 8

7.4

Promised cofinancing

Actual cofinancing

6.6

6 5

4.7 4.7

3.5

3 2.1 2

1.5

2.4

1.5 1.6

1.3

3.2

2.6 1.9

1 0

2002

(N = 11)

2003

(N = 7)

2004

(N = 30)

2005

(N = 23)

2006

(N = 47)

2007

(N = 39)

All years (N = 118)

FY of terminal evaluation report submission Note: Data for 2004 include the World Bank’s multimillion-dollar China Sichuan Gas Transmission and Distribution Rehabilitation Project (GEF ID 75).

For the FY 2007 cohort, a higher level of cofinancing was both promised and reported to have materialized for World Bank projects ($4.10 and $3.80, respectively, per GEF dollar) than for those of the other Implementing Agencies (see table 4.2). However, there are substantial differences among the Agencies in terms of (1) the nature of the projects undertaken, some of which may be easier to raise contributions for than others; and (2) the type of cofinancing raised—loans to the host country (which are relatively easy to raise) versus grants from other donors (which take more effort to obtain and do not require repayment). These differences introduce complexities in the analysis of materialization of cofinancing and will be explored further in future APRs so that a more well-grounded comparison of the performance of the IAs in raising cofinancing is possible. 4. Factors Affecting Attainment of Project Results

Following up on the discussion presented in the FY 2006 APR, the Evaluation Office reviewed the evidence presented in the terminal evaluation reports submitted during FY 2007 to explore the causal relationships between cofinancing and project processes and results. Both causes that are internal and/or external to the project could be responsible for lower than expected levels of cofinancing materialization. The most common internal cause reported for this year’s cohort was inability of projects to show progress in earlier stages of their implementation, resulting in withdrawal of support by non-GEF donors. For example, the non-GEF donors did not fulfill their commitments to as the World Bank’s Renewable Energy and Forest Conservation project in Nicaragua (GEF ID 847), UNDP’s Pilot Production and Commercial Dissemination of Solar Cooker 25

Table 4.2

Materialization of Cofinancing by GEF Agency, 2002–07 Factor

UNDP

UNEP

World Bank

MultiAgency

Total

Number of projects with cofinancing data available

157

Data obtained from terminal evaluation reports

151

118

3.3

2.7

6.2

Data otherwise obtained Approved GEF grant per project (million $)

20.2

5.0

Actual GEF grant per project (million $)

3.2

2.7

5.5

20.2

4.5

Promised cofinancing per project (million $)

5.0

4.4

25.4

134.8

17.4

Promised cofinancing per $1.00 of approved GEF grant

1.50

1.60

Reported materialized cofinancing per project (million $)

4.2

4.1

Reported materialized cofinancing per $1.00 of approved GEF grant

1.30

1.50

Materialized cofinancing per $1.00 of promised cofinancing (%)

4.10 23.7 3.80 94

6.70 103.6 5.10 77

3.50 16.0 3.2 92

project in South Africa, and the World Bank’s global Solar Development Group project because these projects had not made adequate progress during their early stages of implementation. An endogenous cause that affected materialization of cofinancing was savings made during project implementation. For example, for the World Bank’s Renewable Energy for Agriculture project in Mexico (GEF ID 643), savings were made during implementation of project components. Although these savings were passed on to other donors, they were not reflected in lower usage of GEF resources.

Some completed projects reported greater than expected cofinancing to have materialized. Reasons for better than expected cofinancing materialization include, among others, donors contributing higher than expected amounts for a particular project component. In some instances, fluctuations in international currency markets also lead to higher contributions from donors that contribute in nondollar denominations; this was the case for UNEP’s Communitybased Management of On-Farm Plant Genetic Resources in Arid and Semi-Arid Areas of SubSaharan Africa project (GEF ID 981).

An important exogenous cause for lower realization of cofinancing is financial crisis in the host country. For example, for projects such as the World Bank’s Energy Efficiency Project, Sustainability of the National System of Protected Areas project in Bolivia (GEF ID 620), and Landfill Methane Recovery Demonstration Project in Uruguay (GEF ID 766), the host governments did not meet their commitments as a result of a financial crisis faced by their country. In other instances where materialization of cofinancing was lower than expected, the causal variables were not discussed in the terminal evaluation report.

The terminal evaluation reports still do not provide sufficient evidence to allow for an assessment of the extent to which cofinancing was essential in a project’s achieving its global environmental results. This prohibits assessment of the extent to which the reported materialized cofinancing is consistent with the manner in which it is defined by the GEF.

4.2 Delays in Project Completion The Evaluation Office measured the time difference between expected closing at project start and GEF Annual Performance Report 2007

actual closing. The average project completion delay for the FY 2007 cohort was 13 months. This was the same figure as for the FY 2006 cohort; the average delay for FY 2005 was 17 months.1 When considering all 140 projects from the FYs 2007, 2006, and 2005 cohorts for which data on completion delays are available, GEF projects average a 15-month delay in completion. UNDP projects have significantly longer delays than those for the other Agencies, averaging 21 months from scheduled to actual completion (see figure 4.2a). Climate change projects, with an average delay of 22 months between planned and actual completion, experience significantly longer delays compared to those in the other focal areas (see figure 4.2b). Full-size projects were, on average, Figure 4.2

Average Delay in Project Completion by GEF Agency and Focal Area, FYs 2005–07 a. By Agency FSPs All projects MSPs

UNEP (N = 22) World Bank (N = 69) All Agencies (N = 140) UNDP (N = 49)

b. By focal area Biodiversity (N = 77) All focal areas (N = 140) Int’l waters (N = 17)

Climate change (N = 34) 0

Average delay (in months)

4. Factors Affecting Attainment of Project Results

completed after a delay of 19 months; the average delay for medium-size projects was 9 months. Following up on the discussion presented in the FY 2006 APR, the Evaluation Office reviewed the evidence presented in the terminal evaluation reports submitted in FY 2007 to explore the causal relationships of completion delays and project processes and results. Review of the causes for delay shows that while contextual factors are important, delays are in most cases primarily due to internal factors such as overly optimistic project design, communication problems among project partners, delays in transfer of money to the partners, cumbersome implementation procedures, and delays in hiring key staff. Delayed transfer of funds to project partners was reported as a cause of delay for four projects. For the Renewable Energy and Forest Conservation Project in Nicaragua, this occurred because the recipient organization was not well versed in World Bank disbursement procedures. The UNEP–United Nations Industrial Development Organization global project Fostering Active and Effective Civil Society Participation in Preparations for Implementation of the Stockholm Convention (GEF ID 2067) experienced delays in funds transfer because of communication problems between the GEF Agencies and the executing agency. For the other two projects—the World Bank’s Mongolian Dynamics of Biodiversity Loss and Permafrost Melt in Lake Hovsgol National Park Project (GEF ID 984) and its global Coral Reef Monitoring Network Project (GEF ID 814)—the terminal evaluation reports did not explain the reasons behind the delay in transfer of funds. Delays in accomplishment of critical activities may stall progress in accomplishment of other activities. In many instances, projects experience completion delays because of their inability to hire key staff in time, thus potentially delaying 27

the project activities contingent on this hiring. UNDP’s Conservation of Globally Significant Forest Ecosystems in Suriname’s Guyana Shield Project (GEF ID 661) faced delays for this reason; thus, the management team skills required to implement important components of the project were not available in a timely manner. In the case of the World Bank’s Lake Hovsgol National Park Project, an inhospitable climate in the project area made it difficult to undertake project activities during the winter, necessitating a delay in accomplishing the activities scheduled for the summer months. The design of some of the reviewed projects inadequately addressed the complexities involved in implementation. Often, projects underestimated the time it would take to consult with stakeholders, communicate with government agencies and local communities, and resolve local conflicts. For example, the World Bank’s Central Asia Transboundary Biodiversity Project experienced a twoyear completion delay while the project struggled to begin operations in three different countries, each with its own set of complex and cumbersome implementation procedures. The project design of UNDP’s Conservation of Wetland and Coastal Ecosystems in the Mediterranean Region Project (GEF ID 410) was reported to have poor problem analysis and an outdated logical framework, leading to difficulties in project implementation. Delays during project implementation may have negative consequences such as overall inefficiency due to the allocation of a greater proportion of resources to administrative costs rather than project activities, an inability to accomplish some of the project deliverables, a reduction in project scope, lower accumulation of project benefits, and decreased participation by project partners. However, taking additional time for project implementation need not imply poorer performance. In some instances, additional time taken 28

may actually lead to improved performance and results. For example, due to efficient management of its Biodiversity Conservation and Integration of Traditional Knowledge on Medicinal Plants in National Primary Health Care Policy (GEF ID 1410) regional project, UNEP was able to realize savings. Extension of the project duration allowed it to use the savings to undertake other relevant activities geared toward achieving global environmental benefits. Similarly, although persistent conflicts had made the executing agency unable to undertake planned activities in the municipal area initially selected for implementation of UNDP’s Economic and Cost-Effective Use of Wood Waste for Municipal Heating Systems Project in Latvia (GEF ID 914), increasing the project’s duration allowed it to use resources to extend implementation of project activities to other communities that were equally suitable as a target for the project— and that exhibited no conflicts. External causes of project delay include unanticipated political instability, changes in project processing requirements on the part of the host government, and financial crisis in the host country. For example, insurgency in the project area made timely implementation of UNDP’s Upper Mustang Biodiversity Project in Nepal (GEF ID 30) difficult. A completion delay in the World Bank’s Developing the Legal and Regulatory Framework for Wind Power in Russia (GEF ID 2194) was occasioned by a reshuffling of staff and reorganization of Russian agencies, which made it difficult for the project team to negotiate and collaborate with key government staff. The World Bank’s Conservation of Pu Luong-Cuc Phuong Limestone Landscape Project in Vietnam (GEF ID 1477) experienced an 18-month delay primarily due to the new project processing requirements of the host country, which made synchronization of GEF-supported activities with those supported by other donors difficult. GEF Annual Performance Report 2007

Financial crisis may prevent host governments from meeting their cofinancing commitments in a timely manner, leading to delays in project component start-up. Depending on the severity and extent of the crisis, projects may also be restructured because some components and activities may no longer be viable; such restructuring may add to the project’s total duration. The World Bank’s Energy Efficiency Project and its Landfill Methane Recovery Demonstration Project faced delays due to the financial crises that affected their respective host countries. In UNEP’s Community-Based Management of On-Farm Plant Genetic Resources in Arid and Semi-Arid Areas of Sub-Saharan Africa

4. Factors Affecting Attainment of Project Results

Project, one of the host governments (Zimbabwe), which was facing a foreign exchange crisis, insisted on applying an unfavorable exchange rate for converting the GEF grant into local currency. It took time for the project management to negotiate with the Zimbabwe government and convince it to agree to a more favorable rate.

Note 1. The figures reported here differ slightly from those reported in the 2005 and 2006 APRs. The difference derives from the inclusion of projects that were completed ahead of schedule in the analysis for this APR. Such projects are here treated as having been completed without any delay (on time).

5.â&#x20AC;&#x201A; Evaluation of Capacity Development In FYâ&#x20AC;Ż2007, the GEF Evaluation Office began work on the evaluation of GEF capacity development activities. To this end, the Office has completed country case studies of the Philippines and Vietnam, and has reviewed terminal evaluation reports submitted during FYâ&#x20AC;Ż2007. The country case studies examined the nature and results of national, regional, and global interventions and related these to capacity development targets at the policy, institutional, and individual levels in each country. These case studies have been published on the GEF Evaluation Office Web site (www.gefeo.org). The review of the terminal evaluation reports assessed the extent to which capacity development activities in GEF projects are relevant, effective, and efficient and looks at the results and sustainability of the results of these activities based on evidence provided in the reports.

5.1 Findings from the Country Case Studies The country case studies found that the GEF portfolios in the Philippines and Vietnam include considerable capacity development activities. The results are generally positive and, in some areas, significant. GEF capacity development support has been relevant in terms of being in line with national policy priorities and has engendered a good level of national ownership. However, in the Philippines, there is no system that effectively integrates the objectives of capacity development 30

across projects so that an aggregation of impacts can be achieved. The effectiveness of capacity development activities has varied. In Vietnam, some activities were effective in providing new skills and institutional capacities that showed direct and immediate results in the targeted sector; in other cases, the activities had less immediate results, although benefits may develop in the longer term. In many cases in the Philippines, institutions have been unable to provide appropriate incentives for trained staff, and opportunities to use new skills have proved limited. Concerning efficiency, GEF capacity development activities have usually met their immediate output and outcome targets, although a few projects have suffered unusual delays in implementation. The cost-effectiveness of capacity development activities was difficult to assess, since the activities rarely constitute a defined budget heading during project implementation or monitoring. Although there are many improvements to capacity at the individual, institutional, and systemic levels, doubts exist about the sustainability of a number of capacity development outcomes. In the Philippines, several project designs lacked clarity about how the improved capacity will be used, and there are limited incentives available within the government system to retain and reward motivated and trained staff.

In Vietnam, the positive results reflect the substantial efforts put into individual-level capacity development activities in many projects. The changing institutional environment in Vietnam also plays a role in these results, since the need for new skills and approaches is recognized, and improvements to individual capabilities are often accompanied by changes to institutional structures and procedures which mean these capabilities can be utilized. The GEF Evaluation Office found that, both in the Philippines and Vietnam, there has been no systematic monitoring or evaluation of overall capacity development performance at the country level which could promote improvements in coverage or approach. The case studies revealed an underlying weakness in the training programs undertaken by GEF projects, namely a tendency to plan and execute training as a “one-shot” solution. Extensive stakeholder consultations during the evaluation showed the importance of progression and repetition in training. Progression is needed to allow successful trainees who have made use of their new knowledge and skills to undertake more advanced courses and reach higher levels of expertise, thereby further strengthening institutional performance. As for repetition, staff turnover in many government institutions is high, so there is a need to repeat training regularly. One alternative to the one-shot approach would be to tap the existing training resources in a country or region, such as universities or specialist public or private sector training institutions. These can be supported to adapt their existing programs or create new ones to address the key environmentrelated skills identified as necessary during project preparation. In some cases, it may be effective to develop new specialist training entities in a region. One-shot training inputs by international consultants should be a strategy of last resort, when it is evident that the required expertise is 5. Evaluation of Capacity Development

not yet available in the region. In the countries reviewed, the international waters program has been most effective in utilizing and developing regional training capacity. Based on the country case studies, the GEF Evaluation Office delineated further work to help explain the impact of capacity development activities across the GEF portfolio. One component of this work is a meta-evaluation of capacity development findings based on a review of terminal evaluations and previous Evaluation Office reports. The team has also held discussions with the GEF-UNDPUNEP Support Program for Capacity Development on indicators for climate change and has made a presentation on capacity development in climate change at a United Nations Framework Convention on Climate Change workshop. These additional activities will contribute to the development of a set of tools that will enable future APRs, GEF country portfolio evaluations, and the OPS4 to evaluate the achievements of capacity development activities on a broader scale.

5.2 Findings from the Terminal Evaluations Forty-two terminal evaluations, of which 41 were submitted in FY 2007, were reviewed. Of these, 26 are for national projects, 10 for regional projects, and 6 for global projects. The terminal evaluation review identified planned and implemented capacity development activities described in the terminal evaluations and related project documentation. The review analyzed the nature and results of interventions in relation to targets set at the policy, institutional, and individual levels. Figure 5.1 shows types of capacity development activities described in the project documents and terminal evaluations. All 42 projects reviewed included some kind of training activities, whether these were for local executing agencies, other 31

been relevant to national development goals, with the exception of some specific training exercises in five projects. The efficiency of capacity development activities has usually been satisfactory in meeting output and outcome targets. However, 12 projects—or almost 30 percent of the total number reviewed—suffered implementation delays, while 13 were overly ambitious in their goals and targets. Here again, the cost-effectiveness of capacity development activities was difficult to assess, mainly because budget information was only reported for capacity development activities when it was a specific component.

Figure 5.1

Types of Capacity Development Activities in Projects Development of laws, policies, national strategies

57%

Training executing organization staff

79%

Training local communities

50%

Training other organizations

81%

Learning by doing; on-thejob skills development

67%

Awareness-raising/ education activities

98%

Improved resource/enterprise management models

45%

Improved M&E/information systems

76%

Pilot/demonstration activities for wider replication

57%

Establishment of local community/stakeholder groups

36%

Improved production or marketing systems

31%

Improved organizational coordination & communication

67% 0

100

organizations involved in project implementation, or local communities or comprised on-the-job training for individuals and institutions. All but one project included some kind of awareness-raising or educational activity. Thirty-two projects aimed to improve M&E and information systems. More than half of the projects (24) sought to enhance or develop laws, policies, strategies, and standards. The terminal evaluation review findings supported those of the country case studies. The results of capacity development activities have generally been positive. Support has overall 32

The effectiveness of capacity development activities has varied. Of the 24 projects with capacity development activities at the systemic level, 11 projects were successful in improving policies and legal frameworks. The remaining 13 projects were overly ambitious and made little progress or were only able to achieve their aims partially. Five projects created new institutions with varying degrees of outcomes. Concerning training and institutional strengthening, the number of trainings conducted, both formal and on the job, was substantial. Terminal evaluations reported the numbers and types of trainings held and the number of trainees. However, only five, or less than 12 percent, of the terminal evaluations reported on the effectiveness and outcomes of training activities. Although many improvements to capacity at the individual, institutional, and systemic levels were reported in the terminal evaluations, there are concerns about the sustainability of some of the results. A number of changes to policies, legal frameworks, and strategies seem to have a strong level of commitment. Where terminal evaluations reported on the project implementation unit model, it resulted in little or no capacity enhancement of government staff or institutions. GEF Annual Performance Report 2007

The design of these projects did not include a viable project exit strategy. The review of terminal evaluations revealed that the majority of projects considered training as a one-shot exercise. Some projects (seven) established long-term training arrangements, mainly through university-level courses. The review of terminal evaluations found that there is inadequate reporting on the performance of capacity development activities implemented by projects. Few projects included institutional development baselines or assessments, so the intended use of new capacity was unclear. The sustainability of capacity improvements from training remains low because few national or regional long-term training programs were developed. The weak evidence base of capacity development activities makes it difficult to learn lessons or track accountability of project investments.

5.â&#x20AC;&#x192; Evaluation of Capacity Development

5.3 Summary The country case studies and the review of terminal evaluations provide the following lessons on capacity development activities: zz

Capacity development at the systemic level must be realistic. Overly ambitious goals to change policies or laws may, when not attained, adversely affect implementation of other project components.

Institutional strengthening requires baselines and assessments to determine how new capacity will be absorbed.

Long-term training programs and capacities should be based on a national or regional context rather than a dependence on one-shot training.

6. Quality of Project Monitoring Of the 41 terminal evaluation reports reviewed for FY 2007, 33 (80 percent) provided sufficient information to allow the GEF Evaluation Office to rate quality of project monitoring. Of the rated projects, 61 percent were assessed as moderately satisfactory or above for quality of project monitoring. The analysis also confirmed the association between quality at entry of M&E arrangements and quality of project monitoring during implementation.

6.1 Rating Approach Data from 190 terminal evaluation reports—42 from 2004, 41 from 2005, 66 from 2006, and 41 from 2007—submitted by the GEF Agencies to the Evaluation Office were analyzed. Of these reports, 141 provided sufficient information to rate quality of monitoring during project implementation. The Evaluation Office rates quality of project monitoring based on how well a project’s M&E system was implemented. To do so, it assesses whether zz

an M&E system was in place and facilitated timely tracking of results and progress toward project objectives by collecting information on chosen indicators continually throughout the project implementation period; annual project reports were complete and accurate, with well-justified ratings;

the information provided by the M&E system was used for project management;

the parties responsible for M&E activities were properly trained to ensure that correct procedures are followed and quality is maintained in data collection.

6.2 Overall Findings As shown in figures 6.1 and 6.2, the proportion of projects rated moderately satisfactory or above on quality of monitoring was 61 percent for FY 2007. The difference in ratings between the FY 2007 and other cohorts is not statistically significant. Within the FY 2007 cohort, 76 percent of medium-size projects had moderately satisfactory or above ratings, compared to only 44 percent of

Figure 6.1

Percentage of Projects Rated Moderately Satisfactory or Above for Quality of M&E Percentage of projects 100 80 66% 55% 60 40 20 0

2004 (N = 30)

2005 (N = 32)

78% 61%

2006 (N = 46)

2007 (N = 33)

FY of terminal evaluation report submission

6.3 Factors Affecting Quality of Monitoring

Figure 6.2

Quality of M&E during Implementation

Beginning in FY 2006, the GEF Evaluation Office started assessing quality at project entry of M&E arrangements for completed projects and adequacy of funding for M&E activities. This analysis has been developed further and is presented in this APR for FY 2007. Forty projects were rated on the quality of their M&E arrangements at entry. Of these, 27 (68 percent) were rated moderately satisfactory or above in quality of project monitoring during implementation.

Highly satisfactory 3% Unsatisfactory 9% Moderately unsatisfactory 30%

Satisfactory 42%

Moderately satisfactory 15% N = 33 projects

full-size projects. The reasons behind this difference are not clear; it is also not clear whether this is the beginning of a trend. Table 6.1 summarizes ratings on quality of project monitoring by Agency and focal area.

Thirty-two projects were rated both on quality at entry of M&E arrangements and on their quality of monitoring during project implementation. Of these, 24 projects were rated moderately satisfactory or above for quality at entry of M&E arrangements, and 75 percent of these (18 projects) were also assessed as being moderately satisfactory or above for quality of project monitoring during implementation. In contrast, of the remaining eight projects that had been rated as moderately

Table 6.1

Quality of M&E during Project Implementation by GEF Agency and Focal Area Number of projects GEF Agency Rating

UNDP

UNEP

Focal area WB

POPs

Total

Highly satisfactory

Satisfactory

Moderately satisfactory

Subtotal

Moderately unsatisfactory

Unsatisfactory

Highly unsatisfactory

Subtotal

Unable to assess

Total

Note: BD = biodiversity; CC = climate change; IW = international waters; LD = land degradation; MF = multifocal; POPs = persistent organic pollutants.; WB = World Bank.

6. Quality of Project Monitoring

unsatisfactory or below in quality at entry of M&E arrangements, only one was rated as moderately satisfactory or above in quality of project monitoring during implementation. Thus, projects that have a weak M&E plan at entry are unlikely to be able to make sufficient corrections to improve their quality of project monitoring during implementation. It continues to be difficult to determine the adequacy of financial support for M&E activities both in the project budget and during project implementation because a significant proportion of terminal evaluations do not provide sufficient information in this regard. Of the 41 terminal

evaluation reports received in FYâ&#x20AC;Ż2007, only 20 provided enough information to allow an assessment of whether sufficient financial support was provided for M&E activities in the project budget, and only 14 provided sufficient information for actual support during project implementation. Based on the terminal evaluations that provided sufficient information, the GEF Evaluation Office found that for 75â&#x20AC;Żpercent of the projects, sufficient funding had been provided for M&E activities in the project budget and for 64â&#x20AC;Żpercent during implementation. The implications of budgetary support and adequacy of funding on quality of project monitoring are not yet clear.

GEF Annual Performance Report 2007

7. Quality of Terminal Evaluations Terminal evaluations that provide an accurate and candid assessment of project accomplishments and shortcomings are not only essential in forming the building blocks for the APR’s assessment of project outcomes and sustainability but are also a useful learning tool for the GEF partnership. The Evaluation Office reviews terminal evaluations to provide verified ratings on project performance and on the quality of terminal evaluation reports. So far, the ratings verified by the Office have been based on desk review of the terminal evaluation reports, which limits the reliability of these reviews. Field verification enhances the reliability of the reviews by allowing the Evaluation Office to more rigorously confirm the results of completed projects. This year, the Office piloted a methodology for field verification of terminal evaluations for three projects from the FY 2007 cohort. Based on the experience gained in this pilot, the methodology will be further refined, and, beginning next year, the APR will present the findings of field verification of terminal evaluations. To date, 223 terminal evaluation reports have been submitted to the GEF Evaluation Office. This is the fourth year the Office has rated the quality of project terminal evaluation reports; 190 have been rated thus far. In FY 2007, 41 terminal evaluation reports were submitted by the GEF Agencies—a substantial, and thus far, unexplained, drop from

the 66 submitted last year (see figure 7.1). The Evaluation Office tracks submission of terminal evaluation reports based on the expected closing dates of GEF projects. Since many projects face completion delays, this makes for a somewhat unreliable tracking indicator. A better measure would be to track submission based on actual completion of GEF projects. Unfortunately, such information is not now recorded in the GEF database in a timely manner. Often, the operational completion of a project becomes known to the GEF only when its terminal evaluation report has been submitted to the Evaluation Office. The Office will work with the GEF Secretariat and GEF Agencies so that timely submission of terminal evaluation reports for completed projects can be reliably tracked. Figure 7.1

Number of Terminal Evaluation Reports Submitted for Review Number 70 60 50 40 30 20

All projects FSPs MSPs

10 0

2002 2003 2004 2005 2006 2007 FY of terminal evaluation report submission

Overall, the joint efforts of the Evaluation Office and GEF Agencies in improving the quality of terminal evaluation reports seem to be yielding results. Of the 41 terminal evaluation reports submitted this year, the quality of 39 reports (95 percent) was rated to be moderately satisfactory or above.

7.1 Rating Approach The assessment approach adopted for the termi nal evaluation reports submitted in FY 2007 was the same as has been followed since 2005; those submitted in FY 2004 were assessed using a slightly different but comparable methodology. The reports submitted since FY 2005 have been assessed based on the following criteria: zz

Did the report present an assessment of rel evant outcomes and achievement of project objectives in the context of the focal area pro gram indicators, if applicable?

Was the report consistent, the evidence com plete and convincing, and the ratings substanti ated when used?

Did the report present a sound assessment of sustainability of outcomes?

Were the lessons and recommendations sup ported by the evidence presented?

Did the report include the actual project costs (total and per activity) and actual cofinancing used?

Did the report include an assessment of the quality of the project M&E system and its use in project management?

Performance on each of these criteria is rated on a six-point scale. The overall rating is a weighted average of these ratings: the first two criteria are

given a weight of 0.3 each, and the remainder a weight of 0.1 each.

7.2 Findings As noted, of the 41 terminal evaluation reports submitted for review in FY 2007, the quality of 95 percent (39 projects) was rated as moderately satisfactory or above (see table 7.1). By way of comparison, the corresponding proportion of reports so rated in previous years was 84 percent in FY 2006, 88 percent in FY 2005, and 69 percent in FY 2004. For the FY 2007 cohort, the Evaluation Office ratings for quality of terminal evaluation reports were similar for both medium- and full-size projects: for both groups, the quality of 95 percent of the reports was rated as moderately satisfactory or above. To better see the underlying trends in change in quality of terminal evaluation reports, the Evaluation Office has, since FY 2006, compared ratings for quality of terminal evaluation reports based on the year of report completion rather than the year of report submission. However, using year of completion as a basis for comparison has its own limitations. Notably, since all reports for some cohorts—especially the most recent one—have not yet been submitted, ratings for as-yet-unsubmitted reports are not known. Figure 7.2 presents the proportion of terminal evaluation reports with quality rated as moderately satisfactory or above on the basis of year of report completion. The figure shows a clear and sustained trend of quality improvement, with an increasing percentage of terminal evaluation reports being rated as moderately satisfactory or above—rising from 72 percent for reports completed in or before FY 2004, the first year in which the Office began rating report quality.

GEF Annual Performance Report 2007

Table 7.1

Quality of Terminal Evaluation Reports by GEF Agency and Focal Area Number of reports GEF Agency Rating

UNDP

UNEP

Focal area WB

POPs

Total

Highly satisfactory

Satisfactory

Moderately satisfactory

Subtotal Moderately unsatisfactory

Unsatisfactory

Highly unsatisfactory

Subtotal

Unable to assess

Total

Note: BD = biodiversity; CC = climate change; IW = international waters; LD = land degradation; MF = multifocal; POPs = persistent organic pollutants; WB = World Bank.

Performance by Agency For the terminal evaluation reports completed in or before FY 2004, the quality of a substantial percentage of reports from both UNEP and UNDP was in the unsatisfactory range (50 and 40 percent, respectively). Consequently, the FY 2004 Figure 7.2

Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above Percentage 100 80 60 FSPs All projects MSPs

40 20 0

2003 (N = 17)

2004 (N = 46)

2005 (N = 56)

2006 (N = 41)

2007 (N = 23)

FY of terminal evaluation completion Note: The dotted lines imply that the cohorts for those years are not yet complete.

7. Quality of Terminal Evaluations

APR stressed the need for these Agencies to take remedial measures, and the GEF Council at its June 2005 meeting asked UNDP and UNEP to put in place terminal evaluation review processes for GEF projects to improve the quality of their terminal evaluations. UNDP and UNEP have accordingly established more rigorous quality control processes for terminal evaluation. The GEF Monitoring and Evaluation Policy (GEF EO 2006), which was negotiated in the second half of 2005, and the “Guidelines for GEF Agencies in Conducting Terminal Evaluations” (GEF EO 2007), which had been under discussion since the first half of 2006, enunciate GEF expectations of the GEF Agencies in terms of quality control for terminal evaluations and provide greater clarity on this issue. These concentrated efforts have led to an overall improvement in quality of terminal evaluation reports: almost all of the terminal evaluation reports submitted by UNDP and UNEP that were completed during FYs 2006 and 2007 have ratings in the satisfactory range (see figure 7.3, which also shows improvement by focal area).

Figure 7.3

Figure 7.4

Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above, by GEF Agency and Focal Area

Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above on Each Quality Dimension

a. By Agency 60%

UNDP (N = 68)

84% 50%

UNEP (N = 22)

71%

85% 89% 88%

72%

90% 84%

79%

Biodiversity (N = 104)

93% 88%

44%

Climate change (N = 44)

82%

40 20

72%

All focal areas (N = 190)

84% 20

94% 100

Percentage of projects Up to 2004

After 2004

Reporting on outcomes Overall Assessment of sustainability Lessons

2004 (N = 22)

2005 (N = 56)

Consistency & completeness Assessment of M&E system Finance information

2006 (N = 41)

2007 (N = 25)

FY of terminal evaluation completion

92%

89% 80% 83%

Int’l waters (N = 24)

All years

Performance by Quality Dimension Although there has been a significant improvement in overall quality of terminal evaluation reports, improvements in performance on various quality dimensions have been uneven (see figure 7.4). While ratings have improved on dimensions such as reporting on project outcomes, sustainability of outcomes, quality of lessons, and consistency and completeness of reporting, there has been little improvement in the quality of financial information. About half of the terminal evaluation reports received continue to provide insufficient information on the financial performance of completed projects. This lack of information prevents the Evaluation Office from assessing the extent to which there were differences in actual funds mobilization and utilization vis-à-vis expectations, the 40

b. By focal area

100

84%

World Bank (N = 90) All Agencies (N = 190)

94%

causes of variation, and the effect of variation on project performance. Quality of financial reporting is not only contingent on how well terminal evaluations are conducted but also on the extent to which the financial management systems of the GEF Agencies are able to provide data to the evaluators in a timely manner. Since terminal evaluations are generally conducted immediately after a project is operationally closed, some of the financial transactions related to the project may not have been accounted for by Agency financial systems. This often makes it difficult for the Agencies to provide financial information to evaluators in a timely manner. Although this information becomes available once a project is financially closed, by that time its terminal evaluation has generally already been completed. Consequently, financial information is often not incorporated in the terminal evaluation report. There are differences among the GEF Agencies in the quality of financial information provided in their terminal evaluation reports. Of the reports submitted to the Office since FY 2005, 70 percent GEF Annual Performance Report 2007

of those from the World Bank provide adequate financial information, compared to only 41 percent from UNEP and 37 percent from UNDP. This finding is not surprising, in that the Bank is a financial institution which thus accords greater attention to financial reporting. The Evaluation Office will work with the GEF Secretariat and GEF Agencies to identify ways in which financial reporting in the terminal evaluation reports can be improved.

7.3 Agency versus Evaluation Office Ratings For many projects whose terminal evaluation reports have been reviewed by the Evaluation Office, ratings on outcomes, sustainability, and M&E systems are also provided by the terminal evaluation reports and independent evaluation units of the respective GEF Agency. Although the methodologies followed by the Agency evaluation units may not be identical to those followed by the GEF Evaluation Office, the criteria on which performance is rated are often the same. The GEF Evaluation Office compares its ratings with those provided by Agency evaluation units and terminal evaluation reports using both an expanded scale (six- or four-point) and a binary scale to determine the extent to which ratings are consistent. The Independent Evaluation Group of the World Bank has been providing ratings on outcomes, sustainability, M&E, and quality of terminal evaluation reports for some World Bank GEF projects. The UNEP independent Evaluation Office, which provided ratings only on quality of terminal evaluation reports for FY 2006, began providing ratings for other parameters as well during the past fiscal year. Similarly, the UNDP Evaluation Office has begun providing ratings on quality of terminal evaluation reports for some reports submitted during FY 2007. 7. Quality of Terminal Evaluations

Of the 41 terminal evaluation reports submitted to the GEF Evaluation Office in FY 2007, outcome ratings have been provided by the evaluation units of the respective Agencies for 12 projects—6 World Bank projects and 6 UNEP projects. Compared to the outcome ratings given by the GEF Evaluation Office, those given by the Agency evaluation units are, on average, 0.25 points higher on a 6-point scale. This difference is not statistically significant. Moreover, on the binary scale, the ratings accorded by the GEF Evaluation Office and Agency evaluation units were identical. The findings for the FY 2006 cohort were similar to those for the FY 2007 cohort. For 13 completed projects in the FY 2007 cohort, ratings on risks to sustainability of outcomes were provided by both the GEF Evaluation Office and the respective Agency evaluation unit. On a binary scale, these ratings deviated for five projects. Further, the Agency evaluation units rated sustainability of outcomes to be moderately likely or above for 7 of the 13 projects, while the GEF Evaluation Office accorded this rating to only 6 projects. The high level of disagreement in ratings on sustainability underscores the differences in approaches adopted by the Office and the Agency evaluation units. The Office will work with the Agency evaluation units to facilitate convergence in the approaches used to provide ratings on this parameter. The Agency evaluation units rated the quality of 28 of the submitted terminal evaluation reports in the FY 2007 cohort. On a binary scale, both the GEF Evaluation Office and the Agency units rated 27 of these in the satisfactory range (moderately satisfactory and above). Ratings differed for two projects. Given the number of observations, this signifies a substantial convergence in the quality of terminal evaluation report ratings provided by the Evaluation Office and Agency evaluation units. 41

8. Assessment of GEF Agency Climate Policies and Guidelines The GEF Evaluation Office commissioned a survey of the policies and guidelines that GEF Agencies are undertaking to address the GHG emissions generated by their internal operations. The objective was to establish a broad picture of the manner in which the GEF Agencies are managing their GHG footprint.

Reduction targets and metrics to track progress

Governance structure that makes climate a priority issue and integrates it into core Agency operations

Employee engagement program

Centralized GHG data management process

8.1 Assessment Approach

Public disclosure

This assessment was primarily based on a review of publicly available information on the GHG footprint policies and guidelines of the 10 GEF Agencies. Additional supplementary information was gathered by requesting some Agencies to provide relevant documents and via phone conversations. Because several of the Agencies reviewed are in the process of defining their approach to addressing operational GHG emissions, some of the information presented here may be outdated or may contain gaps.

Agency policies and guidelines were assessed using the above-listed criteria (see table 8.1). Since the GEF Secretariat and the GEF Evaluation Office fall under the administrative framework of the World Bank Group, they were not independently reviewed for this assessment. For the most part, the GEF follows World Bank policies regarding GHG emissions. The only difference is regarding travel—while the World Bank requires staff to use least-cost options (which often imply indirect flights), GEF staff are allowed to use direct flights to limit the GHG impact of their travel, even though the financial costs may be higher.

A set of climate evaluation criteria was developed in order to review the Agencies’ GHG reduction approaches. Elements of a robust climate program to reduce the environmental impact of internal operations were identified and included in these criteria. The assessment criteria are as follows:

Agency-wide climate policy

Comprehensive energy-efficiency program

Annual GHG inventory

Information was also gathered for each Agency that is engaged in activities to assess and mitigate the climate impacts of their projects, investments, and loans. Although these efforts are not a focus of this assessment, they certainly can provide significant benefits to the climate and are therefore worth noting.

Table 8.1

African Development Bank



Asian Development Bank



European Bank for Reconstruction and Development Food and Agriculture Organization of the United Nations



International Fund for Agricultural Development



UNDP



United Nations Industrial Development Organization



World Bank



Total

The findings show that the GEF Agencies are on track in addressing the GHG emissions of their internal operations, but that most are in an early stage. Some Agencies do have commitments to further address their internal impacts in the future. For example, Agencies under the United Nations umbrella will be working toward reducing their GHG emissions as part of a UN-wide climateneutral target that they plan to meet in the future.

Public disclosure

Centralized GHG data management



UNEP

Table 8.1 summarizes the existence of climate policies and guidelines for the reviewed Agencies. A checkmark for a particular assessment criterion indicates that the Agency has begun to address this criterion and demonstrated commitment to furthering work in this area.



Inter-American Development Bank

8.2 Findings

Employee engagement

Governing strategy

Reduction targets and metrics

Annual GHG inventory

Energy efficiency program

GEF Agency

Agency-wide climate policy

Assessment of GEF Agencies’ Climate Policies



 8

for assessment. Overall, the banks have made more progress in addressing the climate impacts of their internal operations than have the other Agencies. Eight out of the 10 reviewed Agencies are engaged in activities to assess and mitigate the climate impacts of their projects, investments, and loans. Neither the Inter-American Development Bank nor the International Fund for Agricultural Development has initiated such activities thus far.

Among the Agencies reviewed, the European Bank for Reconstruction and Development, the InterAmerican Development Bank, and UNEP have begun to address most of the criteria considered

All Agencies need to advance their efforts to address their internal climate change impacts and further develop processes and systems to track and manage their internal GHG inventory data. The Agencies should first focus their efforts on completing a robust GHG inventory. Once they have assessed their climate risk through such an inventory, they will be well positioned to develop a cost-effective and disciplined approach to reducing their impacts.

8. Assessment of GEF Agency Climate Policies and Initiatives

9. Management Action Record Management action records are used to track the level of adoption of GEF Council decisions taken on the basis of evaluation findings and recommendations. MARs also aim to increase the accountability of GEF management regarding Council decisions on M&E issues. The GEF Council approved the format and procedures for the GEF MAR at its November 2005 meeting and requested that the GEF Evaluation Office prepare an updated MAR to be presented to the Council for review and follow-up on an annual basis.

Not possible to verify yet—verification will have to wait until more data are available or proposals have been further developed

This year’s MAR presents ratings of GEF management and the verification of these ratings by the Evaluation Office. They track management actions on Council decisions based on 12 GEF Evaluation Office documents, 7 of which, listed below, were included in last year’s MAR: “Annual Performance Report 2004” (GEF/ME/ C.25/1, May 2005)

“Role of Local Benefits in Global Environmental Programs” (GEF/ME/C.27/4, October 2005)

“Annual Performance Report 2005” (GEF/ME/ C.28/2/Rev.1, May 2006)

“GEF Country Portfolio Evaluation: Costa Rica” (GEF/ME/C.28/5, May 2006)

“Evaluation of Incremental Cost Assessment” (GEF/ME/C.30/2, November 2006)

Medium—adopted in some operational and policy work, but not to a significant degree in key areas

“Evaluation of the GEF Activity Cycle and Modalities” (GEF/ME/C.30/6, November 2006)

Negligible—no evidence or plan for adoption, or plans and actions for adoption are in a very preliminary stage

“Evaluation of the GEF Support to Biosafety” (GEF/ME/C.28/Inf.1, May 2006)

This year’s MAR also includes the following five documents included for the first time:

The rating categories for the progress of adoption of Council decisions were agreed upon in a consultative process between the GEF Evaluation Office and the GEF Secretariat and Agencies and are as follows: zz

High—fully adopted and fully incorporated into policy, strategy, or operations

Substantial—largely adopted but not fully incorporated into policy, strategy, or operations as yet

N.A.—not applicable

9.1 Rating Approach

“Annual Performance Report 2006” (GEF/ ME/C.31/1, May 2007)

“Country Portfolio Evaluation: Philippines (1992–2007)” (GEF/ME/C.31/3, May 2007)

“GEF Country Portfolio Evaluation: Samoa (1992–2007)” (GEF/ME/C.31/5, May 2007)

“Joint Evaluation of the Small Grants Programme—ExecutiveVersion”(GEF/ME/C.32/2, October 2007)

“GEF Annual Report on Impact 2007—Executive Version” (GEF/ME/C.32/4, October 2007)

9.2 Findings So far, 12 Council decisions have been fully adopted and thus graduated from the MAR. Six of these were adopted during FY 2006, all pertaining to the “Evaluation of the Experience of Executing Agencies under Expanded Opportunities in the GEF” (GEF/ME/C.30/4, November 2006). The other six, based on recommendations from several evaluations, were fully adopted during FY 2007. The current MAR tracks management actions on 46 Council decisions. The GEF Evaluation Office was able to verify the adoption of 41 of these. The decisions that could not yet be verified are mainly

related to project M&E standards and to the quality of supervision. The Evaluation Office will carry out thematic assessments in the future to assess adoption of these Council decisions. The Office rated 18 (44 percent) of the 41 verified decisions as having been adopted by management at high or substantial levels. This represents an improvement from last year when the percentage of decisions so rated was 33. The Evaluation Office and GEF management agreed on the rating on progress of adoption for 51 percent of the Council decisions (21 of 41); for 46 percent of the decisions (19 of 41), the Evaluation Office gave a lower rating on adoption progress than did management. As shown in table 9.1, most disagreement between management and Evaluation Office ratings are for decisions where management has rated the accomplished adoption to be at a high level. Many of the lower ratings given by the Evaluation Office reflect the fact that even though the GEF has taken measures to address these decisions, these measures cannot be considered to demonstrate a high level of achievement yet. They also reflect the fact that some proposals to the Council are yet to be approved by it. If and when the Council does approve these proposals, substantial adoption may have occurred.

Table 9.1

Ratings of GEF Progress in Adopting Council Decisions GEF Evaluation Office rating Management rating

High

Substantial

Medium

Negligible

Not possible to verify yet

Sum of management ratings

High

Substantial

Medium

Negligible

Not possible to verify yet

Sum of Office ratings

9. Management Action Records

Thirty Council decisions from seven GEF Evaluation Office documents were included in both last year’s and this year’s MARs. The Office was able to follow the progress of adoption of 27 of these decisions; 3 were rated as not possible to verify yet. Of these 27, 52 percent (14) showed progress in the level of adoption, for 33 percent there was no progress, and for 15 percent initial gains had been lost (see figure 9.1). Noticeable progress over last year is related to Council decisions requesting increased transparency in the GEF project approval process through an improved management information system. In this regard, the Secretariat has recently launched a new Project Management Information System which includes a country portal to provide information related to GEF projects at the country level. While the Evaluation Office acknowledges this progress, it notes that several issues remain to be addressed before adoption of these decisions can be rated as high. These include the following: zz

There is some uncertainty as to the accuracy of the information included in the system (for example, updating the status of closed projects).

Figure 9.1

Change in Ratings on Adoption of Council Decisions since APR 2006 Assessment

Ratings decreased 15%

No change 33%

Ratings increased 52%

N = 27 decisions

The portal does not include information on whether (and when) a proposal has been submitted to the GEF Secretariat and is awaiting approval.

It is not clear what status should be used if the project identification form (PIF)/project is not approved (rejected, deferred, or so on).

Eight out of the nine Council decisions that showed no change in progress of adoption from last year’s MAR were rated by the GEF Evaluation Office in both years as having a medium level of adoption, but this does not necessarily mean that the GEF has not addressed these issues in the past year. These include five Council recommendations deriving from the biosafety evaluation for which the GEF has prepared the “Program Document for GEF Support to Biosafety during GEF-4,” which was presented to the Council in April 2008. Although GEF management rates the adoption level of these five decisions as high, the Office rating of medium will be changed if and when the Council approves this document. As in last year’s MAR, the Evaluation Office maintains that the Council decision to integrate local benefits in a more systematic way at all stages of the GEF project cycle has not yet been fully adopted. GEF management rates adoption of this decision as high, but the Office concludes that, while the measures listed by GEF management are encouraging, the adoption rate for this decision is still medium. This assessment is based on the fact that the Office believes that the GEF Secretariat cannot verify the quality of this aspect in project design and implementation because it still has no system in place to involve specialist social development expertise in project review processes. Relatedly, adoption of the Council decision requesting that UNEP and UNDP involve social and institutional expertise in their project supervision was rated as negligible by GEF management. GEF Annual Performance Report 2007

Ratings for the progress of adoption of 4 out of 27 Council decisions (15 percent) have been downgraded by the GEF Evaluation Office this year. The first of these decisions requests the GEF to ensure adequate involvement of expertise on social and institutional issues at all levels of the portfolio. Downgrading of the adoption level of this decision is based on the fact that the GEF has yet to implement a system to address this issue. A second downgraded decision relates to the recommendation that further consideration be given to ways to enhance the contribution of GEF Scientific and Technical Advisory Panel reviews in the project review guidelines. The Office rating was downgraded from substantial to medium because annex 11 of the “GEF Operations Manual” draft (available on February 8, 2008) does not mention the panel’s role in project M&E. The third decision is related to the Council request for new operational guidelines to implement the recommendations of the incremental cost assessment evaluation. Although the GEF Secretariat is in the process of developing a new operations manual, both GEF management and the GEF Evaluation Office downgraded the rating of the adoption of this decision to medium in accordance with the established rating criteria. A substantial level of adoption is expected to be achieved once the manual is approved by the GEF Council. The last downgraded decision has to do with the fact that, although representing a reduction in requirements compared to the previous cycle, there is still room for further simplification of the PIF to keep its criteria simple and unambiguous. In this regard, UNDP has reported that it has to answer an increasing number of questions, from both GEF Secretariat and Council members, in relation to the PIF.

9. Management Action Records

The GEF Evaluation Office is broadly in agreement with GEF management ratings on the level of adoption of Council decisions made in 2007. While full adoption of some Council decisions is expected to be achieved in the short term, others—such as those related to improving project M&E (“GEF Annual Report on Impact 2007”), or improving supervision during project implementation (“Annual Performance Report 2006”)—are likely to take more time to be fully adopted. The Small Grants Programme management has started to address various Council decisions from the SGP joint evaluation through the GEF SGP Steering Committee. However, because the country coordinators are not represented on the Steering Committee, there is a risk that, inadvertently, new proposals will not consider country operational issues and perspectives sufficiently. The inclusion of some senior country coordinators on the Steering Committee would allow for the country program perspective to be heard in the discussions that lead to SGP proposals addressing GEF Council decisions. Although all involved parties agree that this monitoring of the levels of adoption of Council decisions is important and relevant, the Evaluation Office believes that this process is becoming more cumbersome each year. Options such as setting timelines for graduation of Council decisions, breaking down GEF Evaluation Office ratings by actor (rather than the rating being an average of compliance by all actors related to a decision), and presenting clusters of Council decisions by issue (rather than by document as is currently done) will be discussed in the future in order to make this tool more user friendly.

10. Performance Matrix The performance matrix provides a summary of the performance of the GEF Implementing Agencies and GEF Secretariat on a variety of parameters (see table 10.1). Although several of these parameters are assessed by the Evaluation Office on an annual basis, to mitigate against fluctuations in performance ratings due to differences in project mix or other idiosyncratic factors, the values presented in the matrix are, depending on the parameter, running averages of two to four years. Of the 13 parameters included in the performance matrix, information has been provided for 9. Note, however, that the information provided for parameter 13, improvement in performance, addresses only 1 of that parameter’s 12 dimensions. In the future, as data for more years become available, it will be possible to track improvement in performance on a greater number of dimensions. See annex D for methodological notes for the performance matrix data reported for this APR.

10.1 Rating Approach Reporting methodology varies by parameter: zz

Four performance parameters—project outcomes, implementation completion delays, materialization of cofinancing, and quality of monitoring and evaluation during project implementation—are being reported as fouryear running averages, as improvements on these parameters are expected to be gradual.

The figures listed in this year’s matrix, however, may not be four-year averages as data are not available for all these parameters for the preceding three years. zz

The figures reported for performance on quality of project terminal evaluation are twoyear running averages, as meaningful changes can be attained in the short run.

Changes in performance are also likely to be gradual for a second set of parameters: quality of supervision and adaptive management, realism of risk assessment and risk management, and quality of project M&E arrangements. Moreover, assessment of performance on these parameters requires intensive thematic appraisals. For the sake of efficiency, the Evaluation Office will take up such appraisals as part of the APR after a two- to three-year interval.

Several items cannot yet be reported on. zz

For assessing performance on project preparation elapsed time, information will be analyzed from the GEF database. So far, the quality of information available from the database is not sufficiently reliable to facilitate such an analysis. When the recent efforts undertaken by the GEF Secretariat to improve the quality of information in the GEF database fructify, such analyses will be taken up and findings presented in the matrix.

Table 10.1

GEF Agencies and Entities Performance Matrix UNDP

UNEP

World Bank

3. Project preparation elapsed time: average number of months required to prepare projects

4. Implementation completion delays: average delay in completion of projects in months

a. Reported materialization of cofinancing per dollar of approved GEF financing

1.3

1.5

3.9

3.2

b. Reported materialization of cofinancing as percentage of promised cofinancing

6. Independence of Agency central evaluation units

7. Independence of terminal evaluations or independent review of terminal evaluations

8. Realism of risk assessment (robustness of project-at-risk systems): percentage of projects rated moderately satisfactory or above in candor and realism in supervision reporting

Parameter

GEF Overall GEF Secretariat Performance

Results 1. Project outcomes: percentage of completed projects with outcomes rated moderately satisfactory or above Processes affecting results 2. Quality of supervision and adaptive management: percentage rated moderately satisfactory or above Efficiency

5. Materialization of cofinancing

Quality of M&E

9. Robustness of program result indicators and tracking tools 10. Quality assurance of project M&E arrangements at entry

11. Quality of project M&E during implementation

12. Quality of project terminal evaluation

a. Project outcomes

b. Improvement in quality of terminal evaluations: on a scale of 1 (low performance) to 4 (high performance)

Quality of Learning 13. Improvement in performance

For three parameters, the Evaluation Office is in the process of developing reporting approaches. zz

Independence of terminal evaluations. The Office will assess the extent to which systems in the partner Agencies are conducive to unbiased and candid terminal evaluations. This will be appraised through the assessment

10. Performance Matrix

of the process followed for conducting terminal evaluations when terminal evaluation verifications are carried out, and it will be based on the interviews of relevant staff and consultants of the partner Agencies. The following criteria to assess independence of terminal evaluations were presented and discussed with the GEF 49

Agencies and Secretariat at the March 2008 interagency meeting: ––

––

Extent to which the recruitment of the evaluator was independent from direct project implementation; particularly important in this regard will be to consider the degree of separation from those responsible for direct project implementation

––

Extent to which the Agency recruited the appropriate evaluator for the project

––

Extent to which the evaluator had adequate resources (budget and time) to carry out the evaluation; in consultation with the GEF Agencies, the Office will develop benchmarks to assess budgets and time for terminal evaluations for different types of projects considering factors such as number of countries, project sites, diversity of project components, and overall project cost

––

Extent to which the M&E system provides access to timely and reliable information

––

Extent to which there were any undue management pressures on the evaluators regarding the evaluation process (site selection, selection of informants, confidentiality during interviews, information disclosure, ratings, and so on)

––

Extent to which the drafting of the terms of reference was independent from project implementation and included the correct technical input; particularly important in this regard will be to consider the degree of separation of the drafting of the terms of reference from those responsible for project implementation

Extent to which the evaluation was subjected to an independent review process that assessed completeness (including the coverage of all relevant outcomes, risks to sustainability, M&E system, actual project

costs, and cofinancing by component) and the consistency of the text of the report and the ratings zz

The independence of evaluation units of GEF partner Agencies. The GEF Evaluation Office has begun a consultation process with the evaluation units of the GEF Agencies so as to define an appropriate way forward on assessing their independence. An initial proposal is to look to devise a way to assess the extent to which the existing institutional arrangements pose risks to the independence of the evaluation units. To assess this, the Office would draw on self-reports and peer reviews carried out in the context of the Evaluation Cooperation Group of the Banks and the United Nations Evaluation Group. The charter and mandate of the various evaluation units will also provide evidence of their degree of independence.

Quality of learning. The matrix will report on quality of learning through an assessment of improvement demonstrated by the GEF Agencies and entities on the other 12 parameters. This section of the matrix will be accompanied by a narrative that explains the areas in which learning has taken place and will identify the specific changes or factors that have contributed to improved performance. This year, ratings on performance improvement have been provided for changes in quality of terminal evaluation reports.

The assessment of robustness of program result indicators and tracking tools will not be reported on in this APR. Given the highly specialized and technical nature of this assessment, the Evaluation Office will look at this parameter as part of the program studies conducted for the OPS4 in FY 2008, enlisting the assistance of appropriate technical experts. The Office has withdrawn two review criteria—project sustainability ratings and GEF Annual Performance Report 2007

the MAR—which had been proposed earlier, as the former is not a performance criterion but a characteristic of a project, and objective comparative assessment is difficult for the latter.

10.2 Findings As noted, information has been provided on 9 of the 13 parameters included in the performance matrix. However, only two findings, which are more important than the others, are highlighted here. zz

The first, which pertains to project outcomes, is a key measure of the performance of the GEF Agencies in implementing projects that generate global environmental benefits. The GEF Council, as noted elsewhere, has established a 75 percent satisfactory projects benchmark so the actual performance against this measure may be noted. The second finding pertains to improvement in performance on quality of terminal evaluation reports. The performance ratings on other parameters pertain to the baseline figures for that parameter. In subsequent years when

10. Performance Matrix

changes in performance on these parameters are tracked, the findings on these parameters will also become important. Based on the review of terminal evaluation reports submitted to the Evaluation Office from FY 2005 to FY 2007, the Office rated outcome achievement (parameter 1) in 81 percent of the projects to be moderately satisfactory and above. Although the rated projects do not fall under the provisions of the fourth GEF replenishment, the percentage is higher than the 75 percent target specified for GEF-4. Among the Implementing Agencies, the percentage of World Bank projects with ratings in the satisfactory range has been significantly higher than the target (87 percent). For parameter 13, improvement in performance, the Evaluation Office this year assessed performance improvement in quality of terminal evaluations. As noted in conclusion 5 of this APR, there has been a substantial improvement in performance on this dimension. In particular, UNEP and UNDP demonstrated considerable improvement, and the quality of a high percentage of terminal evaluation reports submitted by the World Bank continues to be in the satisfactory range.

Annex A. Terminal Evaluation Report Review Guidelines The assessments in the terminal evaluation reviews will be based largely on the information presented in the terminal evaluation report. If insufficient information is presented in a terminal evaluation report to assess a specific issue such as, for example, quality of the project’s monitoring and evaluation system or a specific aspect of sustainability, then the preparer of the terminal evaluation reviews will briefly indicate so in that section and elaborate more if appropriate in the section of the review that addresses quality of report. If the review’s preparer possesses other first-hand information such as, for example, from a field visit to the project, and this information is relevant to the terminal evaluation reviews, then it should be included in the reviews only under the heading “Additional independent information available to the reviewer.” The preparer of the terminal evaluation review will take into account all the independent relevant information when verifying ratings.

A.1 Criteria for Outcome Ratings Based on the information provided in the terminal evaluation report, the terminal evaluation review will make an assessment of the extent to which the project’s major relevant objectives were achieved or are expected to be achieved, relevance of the project results, and the project’s cost-effectiveness.1 The ratings on the outcomes of the project will be based on performance on the following criteria:2

a. Relevance. Were project outcomes consistent with the focal area/operational program strategies and country priorities? Explain. b. Effectiveness. Are project outcomes commensurate with the expected outcomes (as described in the project document) and the problems the project was intended to address (that is, the original or modified project objectives3)? c. Efficiency. Include an assessment of outcomes and impacts in relation to inputs, costs, and implementation times based on the following questions: Was the project cost-effective? How does the project’s cost/time versus outcomes equation compare to that of similar projects? Was the project implementation delayed due to any bureaucratic, administrative, or political problems and did that affect cost-effectiveness? An overall rating will be provided according to the achievement and shortcomings in the three criteria ranging from highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory, and unable to assess. The reviewer of the terminal evaluation will provide a rating under each of the three criteria (relevance, effectiveness, and efficiency) as follows: zz

Highly satisfactory. The project had no shortcomings. 53

Satisfactory. The project had minor shortcomings.

Moderately satisfactory. The project had moderate shortcomings.

Moderately unsatisfactory. The project had significant shortcomings.

Unsatisfactory. The project had major shortcomings.

Highly unsatisfactory. The project had severe shortcomings.

Unable to assess. The reviewer was unable to assess outcomes on this dimension.

The calculation of the overall outcomes score of projects will consider all three criteria, of which relevance and effectiveness are critical. The overall score will be determined satisfying two different conditions: (1) the overall score on project outcomes will have a constraint that it cannot be higher than the lower score among the critical criteria of relevance and effectiveness; and (2) the overall score cannot be higher than the average score of all three criteria calculated using the following formula: Outcomes = (a + b + c) ÷ 3

In case the average score calculated using this formula is lower than or equal to the lower score among the critical criteria of relevance and effectiveness, then the average score will be the overall score. In case the average score is higher than the lower score among the critical criteria of relevance and effectiveness, then the lower score among the critical criteria will be the overall rating. If the rating for the critical dimensions is unable to assess, then the overall rating for outcomes will also be unable to assess.

A.2 Impacts Has the project achieved impacts, or is it likely that outcomes will lead to the expected impacts? 54

Impacts will be understood to include positive and negative, primary and secondary long-term effects produced by a development intervention. They could be produced directly or indirectly and could be intended or unintended. The terminal evaluation review’s preparer will take note of any mention of impacts, especially global environmental benefits, in the terminal evaluation report, including the likelihood that the project outcomes will contribute to their achievement. Negative impacts mentioned in the terminal evaluation report should be noted and recorded in section 2 of the terminal evaluation review template in the subsection on “Issues that require follow-up.” Although project impacts will be described, they will not be rated.

A.3 Criteria for Sustainability Ratings Sustainability will be understood as the likelihood of continuation of project benefits after completion of project implementation (GEF 2000). To assess sustainability, the terminal evaluation reviewer will identify and assess the key risks that could undermine continuation of benefits at the time of the evaluation. Some of these risks might include the absence of or inadequate financial resources, enabling legal framework, commitment from key stakeholders, and enabling economy. The following four types of risk factors will be assessed by the terminal evaluation reviewer to rate the likelihood of sustainability of project outcomes: financial, sociopolitical, institutional framework and governance, and environmental. The following questions provide guidance to assess if the factors are met: zz

Financial resources. What is the likelihood that financial resources will be available to continue the activities that result in the continuation of benefits (income-generating activities) GEF Annual Performance Report 2007

and trends that may indicate that it is likely that in future there will be adequate financial resources for sustaining project outcomes? zz

Sociopolitical. Are there any social or political risks that can undermine the longevity of project outcomes? What is the risk that the level of stakeholder ownership is insufficient to allow for project outcomes/benefits to be sustained? Do the various key stakeholders see it as in their interest that the project benefits continue to flow? Is there sufficient public/stakeholder awareness in support of the long-term objectives of the project? Institutional framework and governance. Do the legal frameworks, policies, and governance structures and processes pose any threat to the continuation of project benefits? While assessing this parameter, consider if the required systems for accountability and transparency, and the required technical know-how, are in place. Environmental. Are there any environmental risks that can undermine the future flow of project environmental benefits? The terminal evaluation should assess whether certain activities in the project area will pose a threat to the sustainability of project outcomes. For example, construction of a dam in a protected area could inundate a sizable area and thereby neutralize the biodiversity-related gains made by the project.

The reviewer will provide a rating under each of the four criteria (financial resources, sociopolitical, institutional, and environmental) as follows: zz

Likely. There are no risks affecting that criterion of sustainability.

Moderately likely. There are moderate risks that affect that criterion of sustainability.

Moderately unlikely. There are significant risks that affect that criterion of sustainability.

Annex A.â&#x20AC;&#x192; Terminal Evaluation Report Review Guidelines

Unlikely. There are severe risks affecting that criterion of sustainability.

Unable to assess. The reviewer was unable to assess risk on this dimension.

Not applicable. This dimension is not applicable to the project.

A number rating of 1â&#x20AC;&#x201C;4 will be provided in each category according to the achievement and shortcomings, with likely = 4, moderately likely = 3, moderately unlikely = 2, unlikely = 1, and not applicableÂ = 0. A rating of unable to assess will be used if the reviewer is unable to assess any aspect of sustainability. In such instances, it may not be possible to assess the overall sustainability. All the risk dimensions of sustainability are critical. Therefore, the overall rating will not be higher than the rating of the lowest rated dimension. For example, if the project has an unlikely rating in either of the dimensions, then its overall rating cannot be higher than unlikely, regardless of whether higher ratings in other dimensions of sustainability produce a higher average.

A.4 Criteria for Assessment of Quality of Project M&E Systems GEF projects are required to develop an M&E plan by the time of work program inclusion, to appropriately budget the M&E plan, and to fully carry out the M&E plan during implementation. Project managers are also expected to use the information generated by the M&E system during project implementation to improve and adapt the project to changing situations. Given the longterm nature of many GEF projects, projects are also encouraged to include long-term monitoring plans that measure results (such as environmental results) after project completion. Terminal evaluation reviews will include an assessment of the achievement and shortcomings of M&E systems. 55

a. M&E design. Projects should have a sound M&E plan to monitor results and track progress in achieving project objectives. An M&E plan should include a baseline (including data, methodology, and so on), SMART (specific, measurable, achievable, realistic, and timely) indicators and data analysis systems, and evaluation studies at specific times to assess results. The time frame for various M&E activities and standards for outputs should have been specified. Questions to guide this assessment include: In retrospect, was the M&E plan at entry practicable and sufficient (sufficient and practical indicators identified; timely baseline; targets created; effective use of data collection; analysis systems including studies and reports; practical organization and logistics in terms of what, who, and when for M&E activities)? b. M&E plan implementation. The M&E system was in place and allowed the timely tracking of results and progress toward project objectives throughout the project. Annual project reports were complete, accurate, and with well-justified ratings. The information provided by the M&E system was used to improve and adapt project performance. An M&E system should be in place with proper training for parties responsible for M&E activities to ensure that data will continue to be collected and used after project closure. Questions to guide this assessment include: Did the project M&E system operate throughout the project? How was M&E information used during the project? Did it allow for tracking of progress toward project objectives? Did the project provide proper training for parties responsible for M&E activities to ensure data will continue to be collected and used after project closure? c. Other questions. These include questions on funding and whether the M&E system was a good practice.

––

Was sufficient funding provided for M&E in the budget included in the project document?

––

Was sufficient and timely funding provided for M&E during project implementation?

––

Can the project M&E system be considered a good practice?

A number rating of 1–6 will be provided for each criterion according to the achievement and shortcomings, with highly satisfactory = 6, satisfactory = 5, moderately satisfactory = 4, moderately unsatisfactory = 3, unsatisfactory = 2, highly unsatisfactory = 1, and unable to assess = no rating. The reviewer of the terminal evaluation will provide a rating under each of the three criteria (M&E design, M&E plan implementation, and M&E properly budgeted and funded) as follows: zz

Highly satisfactory. There were no shortcomings in that criterion of the project M&E system.

Satisfactory. There were minor shortcomings in that criterion of the project M&E system.

Moderately satisfactory. There were moderate shortcomings in that criterion of the project M&E system.

Moderately unsatisfactory. There were significant shortcomings in that criterion of the project M&E system.

Unsatisfactory. There were major shortcomings in that criterion of the project M&E system.

Highly unsatisfactory. There was no project M&E system.

The rating for M&E during implementation will be the overall rating of the M&E system: Rating on the Quality of the Project M&E System = b

GEF Annual Performance Report 2007

A.5 Criteria for Assessment of Quality of Terminal Evaluation Reports The ratings on quality of terminal evaluation reports will be assessed using the following criteria: a. The report presents an assessment of all relevant outcomes and achievement of project objectives in the context of the focal area program indicators if applicable. b. The report was consistent, the evidence presented was complete and convincing, and ratings were well substantiated. c. The report presented a sound assessment of sustainability of outcomes. d. The lessons and recommendations are supported by the evidence presented and are relevant to the portfolio and future projects. e. The report included the actual project costs (totals, per activity, and per source) and actual cofinancing used. f. The report included an assessment of the quality of the M&E plan at entry, the M&E system used during implementation, and whether the information generated by the M&E system was used for project management. A number rating of 1–6 will be provided for each criterion, with highly satisfactory = 6, satisfactory = 5, moderately satisfactory = 4, moderately unsatisfactory = 3, unsatisfactory = 2, highly unsatisfactory = 1, and unable to assess = no rating. Each criterion to assess the quality of the terminal evaluation report will be rated as follows: zz

Highly satisfactory. There were no shortcomings in the terminal evaluation report on this criterion.

Annex A. Terminal Evaluation Report Review Guidelines

Satisfactory. There were minor shortcomings in the terminal evaluation report on this criterion.

Moderately satisfactory. There were moderate shortcomings in the terminal evaluation report on this criterion.

Moderately unsatisfactory. There were significant shortcomings in the terminal evaluation report on this criterion.

Unsatisfactory. There were major shortcomings in the terminal evaluation report on this criterion.

Highly unsatisfactory. There were severe shortcomings in the terminal evaluation report on this criterion.

The first two criteria (of all relevant outcomes and achievement of project objectives and report consistency and substantiation of claims with proper evidence) are more important and have therefore been assigned a greater weight. The quality of the terminal evaluation reports will be calculated by the following formula: Quality of the Terminal Evaluation Report = 0.3 × (a + b) + 0.1 × (c + d + e + f)

The total number will be rounded and converted to the scale of highly satisfactory to highly unsatisfactory.

A.6 Assessment of Processes Affecting Attainment of Project Outcomes and Sustainability This section of the terminal evaluation review will summarize the factors or processes related to implementation delays and cofinancing that may have affected attainment of project results. This section will summarize the description in the terminal evaluation on key causal linkages of these factors: 57

Cofinancing and project outcomes and sustainability. If there was a difference in the level of expected cofinancing and actual cofinancing, what were the reasons for it? To what extent did materialization of cofinancing affect project outcomes and/or sustainability? What were the causal linkages of these effects? Delays and project outcomes and sustainability. If there were delays, what were the reasons for them? To what extent did the delay affect project outcomes and/or sustainability? What were the causal linkages of these effects?

Notes 1. Objectives are the intended physical, financial, institutional, social, environmental, or other devel-

opment results to which a project or program is expected to contribute (OECD DAC 2002). 2. Outcomes are the likely or achieved short-term and medium-term effects of an interventionâ&#x20AC;&#x2122;s outputs. Outputs are the products, capital goods, and services that result from a development intervention; these may also include changes resulting from the intervention that are relevant to the achievement of outcomes (OECD DAC 2002). For the GEF, environmental outcomes are the main focus. 3. The GEF Secretariat and IAs are currently seeking to better align the focal area program indicators and tracking tools with focal area strategic priorities and project objectives. This will enable the aggregation of outcomes and impacts for each focal area to annually measure progress toward targets in the program indicators and strategic priorities.

GEF Annual Performance Report 2007

Annex B. Terminal Evaluations Reviewed during FY 2007

GEF ID #

Region

Philippines - Samar Island Biodiversity Project: Conservation and Sustainable Use of the Biodiversity of a Forested Protected Area

Asia

FSP

UNDP

5.76

Vietnam - Hon Mun Marine Protected Area Pilot Project

Asia

MSP

0.97

Nepal - Upper Mustang Biodiversity Project

Asia

MSP

UNDP

0.75

110

Regional - Central Asia Transboundary Biodiversity Project

ECA

FSP

10.15

128

Brazil - Energy Efficiency Project

LAC

FSP

15.00

245

Lesotho - Conserving Mountain Biodiversity in Southern Lesotho

Africa

FSP

UNDP

2.51

264

Syria - Supply-Side Efficiency and Energy Conservation and Planning

Asia

FSP

UNDP

4.61

284

Romania - Capacity Building for GHG Emission Reduction through Energy Efficiency

ECA

FSP

UNDP

2.27

410

Global - Conservation of Wetland and Coastal Ecosystems in the Global Mediterranean Region

FSP

UNDP

13.42

445

China - Barrier Removal for the Widespread Commercialization of Energy-Efficient CFC-Free Refrigerators in China

Asia

FSP

UNDP

9.86

505

Pakistan - Mountain Areas Conservancy Project

Asia

FSP

UNDP

10.10

573

Kenya - Removal of Barriers to Energy Conservation and Energy Efficiency in Small and Medium Scale Enterprises

Africa

FSP

UNDP

3.19

595

Global - Solar Development Group

Global

FSP

10.00

597

Regional - Building Partnerships for the Environmental Protection and Management of the East Asian Seas

Asia

FSP

UNDP

16.22

618

Bangladesh - Aquatic Biodiversity Conservation

Asia

FSP

5.00

620

Bolivia - Sustainability of the National System of Protected Areas

LAC

FSP

15.00

643

Mexico - Renewable Energy for Agriculture

LAC

FSP

8.90

661

Suriname - Conservation of Globally Significant Forest Ecosystems in Suriname’s Guayana Shield

LAC

FSP

UNDP

9.54

766

Uruguay - Landfill Methane Recovery Demonstration Project

LAC

MSP

0.98

770

Global - Millennium Ecosystem Assessment

Global

FSP

UNEP

6.96

Project name

Project size

GEF Agency

Approved GEF grant (million $)

Focal area

GEF ID #

Region

801

Slovak Republic - Central European Grasslands - Conservation and Sustainable Use

ECA

MSP

0.75

814

Regional - Coral Reef Monitoring Network in Member States of the Indian Ocean Commission, within the Global Reef Monitoring Network

Africa

MSP

0.74

836

Global - Critical Ecosystems Partnership Fund

Global

FSP

25.00

847

Nicaragua - Renewable Energy and Forest Conservation: Sustainable Harvest and Processing of Coffee and Allspice

LAC

MSP

0.75

913

Philippines - Biodiversity Conservation and Management of the Bohol Islands Marine Triangle

Asia

MSP

UNDP

0.74

914

Latvia - Economic and Cost-effective Use of Wood Waste for Municipal Heating Systems

ECA

MSP

UNDP

0.75

981

Regional - Community-based Management of On-farm Plant Genetic Resources in Arid and Semi-arid Areas of Sub-Saharan Africa

Africa

MSP

UNEP

0.75

984

Mongolia - Dynamics of Biodiversity Loss and Permafrost Melt in Lake Hovsgol National Park

Asia

MSP

0.83

1058

Regional - Pacific Islands Renewable Energy Programme

Asia

MSP

UNDP

0.70

1261

Papua New Guinea - Community-based Coastal and Marine Conservation in the Milne Bay Province

Asia

FSP

UNDP

3.55

1311

South Africa - Pilot Production and Commercial Dissemination of Solar Cookers

Africa

MSP

UNDP

0.80

1325

Regional - Institutional Strengthening and Resource Mobilization for Mainstreaming Integrated Land and Water Management Approaches into Development Programs in Africa

Africa

MSP

1.00

1384

Global - Biodiversity Indicators for National Use

Global

MSP

UNEP

0.82

1410

Regional - Biodiversity Conservation and Integration of Traditional Knowledge on Medicinal Plants in National Primary Health Care Policy in Central America and Caribbean

LAC

MSP

UNEP

0.75

1477

Vietnam - Conservation of Pu Luong-Cuc Phuong Limestone Landscape

Asia

MSP

0.75

1646

Russian Federation - Cost Effective Energy Efficiency Measures in the Russian Educational Sector

ECA

MSP

UNDP

1.00

1851

Regional - Protection of the North West Sahara Aquifer System and Related Humid Zones and Ecosystems

Africa

MSP

UNEP

0.60

2067

Global - Fostering Active and Effective Civil Society Participation in Preparations for Implementation of the Stockholm Convention

Global

POP

MSP

UNEP

1.00

2194

Russian Federation - Developing the Legal and Regulatory Framework for Wind Power in Russia

ECA

MSP

0.73

2344

Regional - Desert Margins Programme Tranche 2 - Africa

Africa

FSP

UNEP

5.62

3036

Regional - Supporting Capacity Building for the Third National Reporting to CRIC-5/COP-8 (Asia Pacific)

Asia

MSP

UNDP

0.51

Project name

Project size

GEF Agency

Approved GEF grant (million $)

Focal area

Note: BD = biodiversity; CC = climate change; ECA = Europe and Central Asia; IW = international waters; LAC = Latin America and the Caribbean; MF = multifocal; POP = persistent organic pollutant; WB = World Bank.

GEF Annual Performance Report 2007

Annex C. Greenhouse Gas Footprint Policies and Guidelines Assessment C.1 Criteria The criteria for greenhouse gas footprint policies and guidelines assessment include the following.

––

Regularly scheduled energy audits to find inefficiencies and areas for improvement with focus on largest facilities

––

Building benchmarking to compare efficiency of buildings to others within the organization and available industry standards

––

Performance incentives for energy and facility managers to meet energy efficiency goals

––

Capability to assess cost-effectiveness of energy efficiency investments

––

A commitment to invest in all no-cost or low-cost efficiency measures

Agency-Wide Climate Policy zz

Does the Agency’s climate policy contain and communicate the Agency’s mission and core values with respect to climate change?

Does it contain and communicate commitments to control and improve its carbon impact and environmental performance with respect to significant climate aspects of the organization’s products, activities, and/or services?

Does it contain and communicate how the Agency plans to address climate impacts of internal operations? Does the Agency have a policy as to whether to evaluate its lending portfolio and investments based on climate criteria?

Annual GHG Inventory zz

Is the Agency’s GHG inventory consistent with the GHG Protocol developed by the World Resources Institute and the World Business Council for Sustainable Development?

Does it include an inventory management plan to make inventory processes repeatable and transparent?

Does it include at a minimum, all scope 1, scope 2, business travel emissions (scope 3), and emissions from the hosting of conferences (scope 3)?

Does it include all employees and on-site consultants?

Does it include headquarters, regional offices, and field offices?

Comprehensive Energy Efficiency Program zz

Does the Agency have an energy efficiency program that includes improvements in buildings, equipment, and lighting? The program should include all forms of energy use at the Agency, including purchased electricity, on-site fuel use, and use of vehicles. Does the Agency have the following elements of an energy efficiency program:

Are results made public through an annual report, Web site, or other venue?

Is the inventory signed off on by an Agency senior executive?

them as to what they can do at work and at home to reduce their impact? zz

Are tools available to collect and share best practices in areas such as business travel, equipment management and support, food services, mail and transportation services, and supplier management?

Are incentive programs in place to encourage behaviors that reduce GHG impacts at home and in the office?

Does the Agency provide employee commuter benefits?

Reduction Targets and Metrics to Track Progress zz

Has the Agency set an absolute GHG reduction target and energy efficiency targets over a 5- to 10-year time period?

Has the Agency set normalized targets for energy efficiency and GHG reductions such as emissions/square foot and emissions per employee?

Governance structure That Makes climate a Priority Issue and Integrates It into Core Agency Operations zz

Is the Agency’s board of directors actively engaged in climate change policy and has it assigned oversight responsibility to a board member, board committee, or full board? Does the Agency president assume a leadership role in articulating and executing climate change policy? Are top executives and/or executive committees assigned to manage climate change response strategies? Does the climate management team review each year the Agency’s climate change program to ensure its continuing suitability, adequacy, and effectiveness? ––

Is executive officers’ performance evaluation linked to attainment of climate strategy goals and GHG targets?

Employee Engagement Program zz

Are there training programs in place to educate staff about climate change and to empower

Centralized GHG Data Management Process zz

Are corporate-level processes in place for collecting and calculating GHG information to ensure consistency and completeness?

Are tools available for analyzing GHG data to create prioritized lists of actionable reduction initiatives?

Public Disclosure zz

Does the Agency publicly disclose material risks and opportunities posed by climate change?

Do the communications to the public offer comprehensive, transparent presentation of response measures?

C.2 Findings on Individual Agencies The following table summarizes the status of GHG footprint policies and guidelines for each of the 10 GEF Agencies. A checkmark for a particular assessment criterion indicates that the Agency has begun to address this criterion and has demonstrated commitment to furthering work in this area.

GEF Annual Performance Report 2007

Table C.1

Status of Agency GHG Policies and Guidelines African Development Bank (AfDB) Agency-wide climate policy



Formulation of a comprehensive corporate environmental strategy that includes GHG impacts is currently under development.

Energy efficiency program



As part of AFDB’s corporate environmental strategy, a comprehensive energy efficiency program was developed.

Annual GHG inventory



A comprehensive GHG inventory was recently completed.

Reduction targets and metrics Governing strategy

No information could be found about AfDB reduction targets or metrics for climate change. 

AfDB is creating an Environmental Working Group that will consist of several experts and board members who will review and assess implementation of the environmental strategy as practiced throughout AfDB. GHG emissions are a component of the strategy.

Employee engagement program

No information could be found about AfDB’s employee engagement program.

Centralized GHG data management process

No information could be found about a centralized GHG data management process.

Public disclosure

The AfDB Web site does not contain climate program information, and this information was not found through any other public source.

Initiatives to assess and mitigate impacts of loans, projects, and investments

AFDB is currently developing a strategy aimed at integrating climate risk management and adaptation into its operations. The goal of the strategy is to safeguard and enhance the effectiveness of AfDB investments in a changing climate, and to improve its support to its regional member countries in the area of climate risk management and adaptation.

Information sources

www.afdb.org; AfDB Corporate Environmental Initiatives Analysis and Strategy Framework document, January 2008; Bank Group Climate Risk Management and Adaptation Strategy– Approach Paper, February 2008 Asian Development Bank (ADB)

Agency-wide climate policy



ADB’s climate change brochure, REACH program, and Carbon Market Initiative detail its climate change initiatives. ADB has an energy policy created in 1995 that includes its climate change program, but not a separate policy for climate change initiatives. ADB has an energy strategy under development, which will guide its future operations in the energy sector. The issues the strategy will address are energy security, global warming/climate change, sector policy reform and governance, and energy efficiency. ADB has a robust environmental policy, but it provides little information on how the bank addresses climate change within that policy. The environmental policy deals with five major challenges at ADB; within the third challenge, the Need to Maintain Global and Regional Life-Support Systems, there is some mention of the need to address climate change.

Energy efficiency program



ADB promotes energy efficiency in how its buildings are cooled, heated, ventilated, and lit. Its headquarters building in Manila is fitted with the most efficient technologies available today (www.adb.org/documents/brochures/ADB-energy-efficiency.pdf).

Annual GHG inventory



The AfDB paper on its environmental strategy references the results of an ADB GHG inventory.

Reduction targets and metrics Governing strategy

Employee engagement program

No information could be found about ADB reduction targets and metrics for climate change. 

ADB’s president, Haruhiko Kuroda, is actively engaged in the climate change issue. His remarks are contained in the foreword of the ADB’s climate change brochure developed in 2007. There is no information available about ADB’s employee engagement efforts.

Annex C. Greenhouse Gas Footprint Policies and Initiatives Assessment

Asian Development Bank (ADB) Centralized GHG data management process Public disclosure

No information could be found about the ADB GHG data management process. 

The ADB Web site links to all its initiatives and programs to mitigate climate change.

Initiatives to assess and mitigate impacts of loans, projects, and investments

In 2005, ADB established the Energy Efficiency Initiative with the aim of compiling and analyzing existing knowledge and experience on energy efficiency policies and formulating a clean energy investment strategy with an indicative lending target of $1 billion over the next few years. The Carbon Market Initiative, approved by the ADB Board in November 2006, provides financial and technical support to developers and sponsors of projects with GHG mitigation benefits that can qualiFY as an eligible clean development mechanism project under the Kyoto Protocol.

Information sources

www.adb.org/Documents/Policies/Environment/env020100.aspwww.adb.org/Documents/ Books/sustainability-report/default.asp; www.adb.org/RSDD/default.asp; www.adb.org/ Clean-Energy/default.asp; www.adb.org/Documents/Brochures/Climate-Change/ Climate-Change-Brochure.pdf; AfDB Corporate Environmental Initiatives Analysis and Strategy Framework, January 2008 European Bank for Reconstruction and Development (EBRD)

Agency-wide climate policy

EBRD does not have a specific climate policy, but it does include climate initiatives in its 2005 energy policy. There is also a brief mention of following Intergovernmental Panel on Climate Change and United Nations Framework Convention on Climate Change mechanisms for climate change in its environmental policy.

Energy efficiency program



EBRD has a number of energy efficiency initiatives under way. It completed an independent study on its water and energy consumption in 2005. EBRD headquarters is currently being refurbished with installation of new lighting and an energy management system to reduce its impact on the environment. The EBRD energy policy guidelines are published online and include several initiatives for energy efficiency (www.mtvsz.hu/dynamic/ ebrdeen.pdf).

Annual GHG inventory



EBRD conducted a preliminary GHG assessment in 2005. It has published guidelines for others to follow its methodology (www.ebrd.com/enviro/tools/ghg.pdf).

Reduction targets and metrics



The energy policy of 2005 set renewable energy targets for EBRD including a target to meet at least 10 percent of its electricity needs by renewable sources. Other reduction goals and achievements include (www.ebrd.com/pubs/general/sus05.pdf): • 2005: reduced paper use to 78.3 tons from 96 tons • Recycles light bulbs, paper, glass, printer and toner cartridges • Employ a refrigerant recovery unit to collect/reuse refrigerants • Use video conferencing for meetings when possible

Governing strategy



EBRD created a multidepartmental working group to consider a range of energy and environment issues, specifically concerning energy efficiency, increasing recycling, and reducing environmental impact. The working group brings together the administration department, environmental department, communications, and staff council members.

Employee engagement program



Many organizations throughout ERBD support energy efficiency and climate initiatives. EBRD encourages staff at its London headquarters to reduce car use by providing interestfree loans for season tickets for public transport. It also provides secure storage space for people who cycle to work (www.ebrd.com/pubs/general/sus05.pdf).

Centralized GHG data management process Public disclosure

No information could be found about an EBRD centralized GHG data management process. 

EBRD discloses its climate change information on its Web site and in its annual sustainability reports. It includes information on internal and external climate projects. EBRD embraces the Global Reporting Initiative reporting elements; these provide guidelines for sustainability reporting initiatives for voluntary use by organizations to report their activities concerning economic, social, and environmental actions from their activities, products, and services (www.ebrd.com/pubs/general/sus05.pdf).

GEF Annual Performance Report 2007

European Bank for Reconstruction and Development (EBRD) Initiatives to assess and mitigate impacts of loans, projects, and investments

The energy policy of 2005 sets a target that 50 percent of all lending in the power sector should be earmarked for increasing energy efficiency. EBRD analyzes its increase in carbon emissions from projects financed. This information was reported in the EBRD Sustainability Report 2005. The direct investments signed by EBRD in 2005 are estimated to lead to a net increase in GHG emissions of 5.6 Mtpa.

Information sources

www.ebrd.com/enviro/index.htm Food and Agricultural Organization of the United Nations (FAO)

Agency-wide climate policy

FAO does not have a stand-alone climate policy, but many of its programs address climate change. It contributes to the climate change debate by assessing the available scientific evidence, participating in observing and monitoring systems, collecting unique global data sets, and providing a neutral forum for negotiations and technical discussions on climate change and agriculture.

Energy efficiency program

No information could be found about FAO’s energy efficiency activities.

Annual GHG inventory

No information could be found about FAO’s annual GHG inventory.

Reduction targets and metrics

No information could be found about FAO’s reduction targets and metrics for climate change.

Governing strategy



It appears that the issue of climate change is integrated into core operations, but it is unclear if there is oversight from a board or top management.

Employee engagement program

No information could be found about FAO’s employee engagement program.

Centralized GHG data management process

No information could be found about FAO’s GHG data management.

Public disclosure

FAO has a good deal of climate information readily available on its Web site, but this does not cover any activities it is implementing to reduce the impact of its internal operations.

Initiatives to assess and mitigate impacts of loans, projects, and investments

In 2001, an integrated climate change program based on current activities was approved. This includes the promotion of practices for climate change mitigation, the adaptation of agricultural systems to climate change, the reduction of emissions from the agricultural sector as far as it is carefully considered within the major objective of ensuring food security, the development of practices aimed at increasing the resilience of agricultural production systems to the vagaries of weather and climate change, national and regional observing systems, as well as data and information collection and dissemination. Through the newly established Environment, Climate change and Bioenergy Division, FAO will strengthen its information and advocacy activities to encourage investment in preparedness; promotion of adaptation of agricultural, forestry and fishery systems to climate change to ensure that efforts to protect the land and aquatic base for sustainable agricultural production are integral elements of climate change adaptation measures. Information and analysis will be critical and so will capacity building.

Information sources

www.fao.org/clim/; www.fao.org/clim/role_en.htm Inter-American Development Bank (IDB)

Agency-wide climate policy



IDB has a Carbon Neutral Initiative and is investing in projects in Latin America and the Caribbean, which is based on four lines of action: calculating emissions, reducing emissions, offsetting emissions, and setting an example in Latin America and the Caribbean. IDB headquarters has been carbon neutral since 2007; in 2008, IDB country offices will also be carbon neutral. The IDB commitment to going carbon neutral supports the objectives of the Environment and Safeguards Compliance Policy and the Sustainable Energy and Climate Change Initiative, both of which encourage the reduction and control of GHG emissions. As part of its commitment, IDB has become a partner of the U.S. Environmental Protection Agency (EPA) for three programs: EPA Climate Leaders, EPA Green Power Partnership, and EPA Energy Star. Full details of IDB’s partnerships with EPA will be available online in April 2008 (http://idbdocs. iadb.org/wsdocs/getdocument.aspx?docnum=932547).

Annex C. Greenhouse Gas Footprint Policies and Initiatives Assessment

Inter-American Development Bank (IDB) Energy efficiency program



As an EPA Energy Star partner, IDB has made a voluntary commitment to measure, track, and benchmark energy performance. In 2007, the IDB main headquarters buildings received an Energy Star rating of 84 out of 100. In addition, IDB has committed to developing and implementing a plan to improve energy performance. To that end, it is continually identifying ways to reduce energy and gas usage and conducting research into improved energy performance and ozone protection. In 2007, this included the purchase of renewable energy certificates equivalent to 100 percent of its energy consumption in its four U.S. headquarters buildings, a move recognized by IDB acceptance into the EPA Green Power Partnership. In 2008, IDB intends to begin energy audit of its buildings (www.iadb.org/sds/ reports/sustainability2006/Commitments.cfm?language=en).

Annual GHG inventory



IDB conducts an annual GHG inventory for its headquarters emissions. This included all direct emissions associated with the operation of IDB offices, such as heating and electricity, chiller emissions; and indirect emissions from IDB travel (air travel and hotel nights). In 2007, these emissions were approximately 22,000 tCO2eq. In addition, in 2007 IDB began to conduct individual inventories for each of its 26 country offices and 2 nonregional offices. The calculation of these is in its final stage. Results are expected in April 2008 and will be made public. In conducting its GHG inventory, IDB follows the EPA’s Climate Leaders Greenhouse Gas Inventory Protocol, Design Principals, and the GHG Protocol. IDB also calculates emissions associated with its annual meeting of the Board of Governors. Full details of the Bank’s GHG inventory will be available online in April 2008 (http://idbdocs.iadb.org/ wsdocs/getdocument.aspx?docnum=932547). IDB is currently developing an inventory management plan, which sets out a clear approach to managing and monitoring GHG data. A decision has not yet been taken as to whether this will be a public document.

Reduction targets and metrics



To meet a goal of carbon neutrality, IDB annually offsets carbon and other emissions that cannot be reduced by greening and efficiency measures and programs. It purchases verified emission reductions (VERs) from leading carbon brokerage firms, ensuring credibility and delivery guarantee; these are sourced from offset projects in Latin America and the Caribbean, focusing on energy efficiency, renewable energy, and fuel switch projects which provide additional environmental and social benefits for local communities. In 2007, this included projects in Peru, Guatemala, and Colombia. In 2008, carbon offsets will be purchased to ensure that country offices are also carbon neutral. IDB will continue to purchase renewable energy certificates equivalent to 100 percent of its energy consumption in its four headquarters buildings in the United States, and seek energy efficiency measures both at headquarters and in its country offices. As part of its commitment to EPA climate leaders, IDB is in the process of discussing and setting a GHG reduction goal to be achieved over the next 5 to 10 years.

Governing strategy



In 2006 IDB President Moreno tasked the IDB’s sustainability and facilities functions with forming a Greening the Bank Task Force to address issues of internal sustainability including energy efficiency. In discussing this issue in 2007, he underlined: “We are committed to doing all we can to address the issue of climate change in the region and provide sustainable development opportunities to local communities. We look forward to applying the [carbon neutral] concept to other Bank activities.”

Employee engagement program



In 2008, IDB will be focusing on improving information available internally and externally on the way in which it tackles the challenges of climate change internally. This includes the launch of a new Sustainability Portal, as well as planned workshops to engage staff in the carbon neutral initiative. The current focus is on discussing the initiative with country office staff who are providing inventory data.

Centralized GHG data management process Public disclosure

Information sources

IDB is documenting its inventory process through an inventory management plan. It is unclear if the data collection process has been centralized. 

IDB has numerous links to carbon footprint information on its Web site. A new Sustainability Portal will be launched in April 2008 with updated information on IDB’s greening and carbon neutral initiatives, as well as the 2007 Sustainability Review. www.iadb.org/sds/carbon_neutral/; www.iadb.org/sds/reports/sustainability2006/highlights. cfm?language=English

GEF Annual Performance Report 2007

International Fund for Agricultural Development (IFAD) Agency-wide climate policy



Many of IFAD’s programs take climate initiatives into consideration. IFAD understands the issues surrounding climate change and its mission. For IFAD, climate change has a special significance. Its mission is to enable poor rural people to overcome poverty. Agriculture is the main source of livelihood for most poor rural people, and it is also the human activity most directly affected by climate change. In 1988, an interdepartmental Group on Climate was established by FAO to closely monitor the state of climate change science, develop the position of the organization, and participate in the international discussions, particularly after the adoption of the United Nations Framework Convention on Climate Change (1992) and the Kyoto Protocol (1997).

Energy efficiency program

No evidence of an internal energy efficiency program was found for IFAD.

Annual GHG inventory

No information could be found about IFAD’s annual GHG inventory program.

Reduction targets and metrics

No information could be found about IFAD’s reduction targets or metrics for climate change.

Governing strategy

Lennart Båge, president of IFAD, published, “Climate Change: A Growing Challenge for Development and Poverty Reduction,” indicating that top management is actively engaged on climate initiatives, but there is no evidence the issue is integrated into core operations.

Employee engagement program

There is no public information available about what IFAD does to engage employees on the issue of climate change.

Centralized GHG data management process

No information could be found about IFAD’s GHG data management process.

Public disclosure

IFAD has a good deal of climate change science information readily available on its Web site, although it does not disclose any internal climate activity information.

Information sources

www.ifad.org/climate/index.htm United Nations Development Program (UNDP)

Agency-wide climate policy

UNDP does not have a climate policy to direct climate activities. Even without a policy, it has implemented many climate initiatives (www.energyandenvironment.undp.org/undp/ indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=1019).

Energy efficiency program Annual GHG inventory

No information available on an UNDP internal energy efficiency program. 

Reduction targets and metrics Governing strategy

UNDP’s 2006 GHG inventory estimate data was included in a recently completed United Nations GHG inventory. UNDP has an ongoing process to improve its inventory data and eventually complete a comprehensive GHG inventory. UNDP supports the UN Secretariat’s goal for all UN agencies to be climate neutral, although the Agency has not yet committed to this goal.



UNDP is working across the world to help developing countries build the capacity needed both to adapt to the impacts of climate change and dramatically expand the reach of affordable, improved energy services to the 2 billion people who currently go without. “As global citizens we cannot afford to wait to address these urgent issues,” noted UNDP Administrator Mark Malloch Brown. Two bureau heads are in charge of the Agency’s greening initiatives including its climate-neutral effort.

Employee engagement program

No information was found on internal UNDP employee engagement programs for climate change.

Centralized GHG data management process

No information was found on a UNDP centralized GHG data management process.

Public disclosure

UNDP has information on its climate change initiatives on its Web site. Little information was found on its internal climate activities.

Annex C. Greenhouse Gas Footprint Policies and Initiatives Assessment

United Nations Development Program (UNDP) Initiatives to assess and mitigate impacts of loans, projects, and investments

To mobilize the benefits of carbon financing for the developing world, UNDP established the MDG Carbon Facility, an innovative mechanism for the development and commercialization of emission reduction projects. The core objectives of the facility are (1) broadening access to carbon finance by enabling a wider range of developing countries to participate, particularly those countries that are presently underrepresented; and (2) promoting emission reduction projects that contribute to the UN Millennium Development Goals, yielding additional sustainable development and poverty reduction benefits. The UNDP Initiative for Sustainable Energy provides a policy framework for UNDP support to countries seeking to integrate sustainable energy activities into their national plans for social, economic, and environmental development. Its focus is on encouraging countries to establish appropriate legal, regulatory, and institutional frameworks for increasing access to energy services and financing mechanisms. In 2000, 66 percent of UNDP’s country offices reported working with governments on sustainable energy projects involving energy efficiency, renewable energy, and energy planning.

Information sources

http://hdr.undp.org/en/media/hdr_20072008_summary_english.pdf; www.undp.org/climatechange/; phone conversation with UNDP environment staffer United Nations Environment Program (UNEP)

Agency-wide climate policy



Energy efficiency program

A UNEP climate neutral strategy has been developed with the specific objective of making UNEP climate neutral as of January 1, 2008. The strategy has been endorsed by UNEP’s senior management team in September 2007. No information is available about UNEP’s energy efficiency program.

Annual GHG inventory



An initial inventory of UNEP GHG emissions was prepared in 2007 highlighting air travel and electricity consumption as two major sources of GHG. A Preliminary Comprehensive GHG Emissions Inventory is currently under preparation. It will be finalized by the end of March 2008.

Reduction targets and metrics



UNEP has a goal of becoming climate neutral as of January 1, 2008. A detailed Action Plan for Climate Neutral UNEP is in development based on the analysis of energy use patterns and emissions inventories from the Preliminary Comprehensive GHG Emissions Inventory.

Governing strategy



The UNEP/GEF document “Investing in Our Planet” notes the following: “On Climate Change, UNEP’s work builds on its expertise in assessment, standard setting, methodology development, and demonstration. UNEP draws upon its own programs, its Risoe Centre and networks such as the International Energy Agency, Global Network for Energy and Sustainable Development, REN21, REEEP, the IPCC, Global Change Observing Systems, WMO and the International Solar Energy Society. Much of UNEP’s work takes on a market sector approach responding to environmental drivers. UNEP works with National Cleaner Production Centers to address energy and chemical use. UNEP has developed disaster preparedness methods through its APELL program, important to climate change adaptation vulnerability through its assessment work.” (http://dgef.unep.org/intranet/strategy/ UNEP-2pager-18April07.doc).

Employee engagement program



A training program on green practices in the office, including activities to reduce GHG emissions, is being prepared in partnership with the United Nations Office in Nairobi. The program will also cover green practices that staff members may implement in their private life outside the office.

Centralized GHG data management process



A new position has been created within UNEP to coordinate and monitor the implementation of the Climate Neutral Strategy. The functions will include (1) design and implement systems to collect and manage GHG data for UNEP headquarters operations and provide advice to UNEP outpost offices on how to design and implement such systems; (2) coordinate with UNEP outpost offices on the collection of GHG data; and (3) develop the UNEP global GHG inventory using the World Resources Institute-World Business Council for Sustainable Development GHG reporting protocol and calculate the UNEP climate footprint, consistent with the UN systemwide Climate Neutral Strategy.

GEF Annual Performance Report 2007

United Nations Environment Program (UNEP) Public disclosure

UNEP has information on its external climate change initiatives on its Web site. Little information is available on its internal climate activities.

Initiatives to assess and mitigate impacts of loans, projects, and investments

The UNEP Risoe Center on Energy, Climate and Sustainable Development supports UNEP in its aim to incorporate environmental aspects into energy planning and policy worldwide, with special emphasis on developing countries. The center supports research by local institutions, coordinates projects, disseminates information, and conducts a full in-house research program in close collaboration with other institutions in Denmark and internationally. It pursues its objectives through initiation of and participation in UNEP-sponsored energy-environment projects at the national or regional level, research and methodological development on energy-environment issues and climate change mitigation, and technical support to programs on energy and climate change.

Information sources

www.unep.org/DEPI/; www.climateneutral.unep.org/cnn_contentdetail. aspx?m=131&amid=194; www.uneprisoe.org/index.htm United Nations Industrial Development Organization (UNIDO)

Agency-wide climate policy



Energy efficiency program

UNIDO has recently set up a basic in-house coordination mechanism, its Climate Team, and is beginning to put the building blocks in place. The Climate Team will shortly be submitting an initial strategy to UNIDO’s Executive Board on how to implement its commitment to climate neutrality. No information is available on UNIDO’s internal energy efficiency program.

Annual GHG inventory



UNIDO completed a rough estimate of its 2006 GHG inventory as part of the UN Environment Management Group’s effort to complete a GHG inventory estimate of all UN agencies.

Reduction targets and metrics



UNIDO is committed to the UN Secretariat’s goal for all UN agencies to reach climate neutrality although a date for meeting this goal has not been set.

Governing strategy



Reducing the impacts of climate change has been integrated into the operations of UNIDO, and UNIDO recently created a Climate Team that will soon present a climate-neutral strategy for approval to the Executive Board.

Employee engagement program

Employees at many divisions within UNIDO work on energy efficiency. There is no indication of an employee engagement program educating staff about how they can reduce their climate impacts.

Centralized GHG data management process

No information could be found about UNIDO’s GHG data management process.

Public disclosure

UNIDO discloses information about its external climate change initiatives to the public. No information is available on its internal climate activities.

Initiatives to assess and mitigate impacts of loans, projects, and investments

UNIDO’s industrial energy efficiency program helps industries and support institutions adopt a systems approach to lower energy consumption, which can lead to huge efficiency gains. UNIDO’s energy efficiency capacity building and advisory services include (1) provision of train-the-trainer interventions with the objective of transferring to national specialists the knowledge and skills required to enable them improve the efficiencies of industrial energy systems; (2) training on financing aspects of energy-efficiency investments to enable trained experts to commercially market their services to industrial clients; (3) creating awareness of the importance of life-cycle costing in energy-efficiency analysis, demonstrating that over their working life motor and steam boiler systems consume energy and fuel costing far more than the initial capital investment for the system; and (4) assistance in promoting the necessary financing and investment for energy efficiency at national and industry levels.

Information sources

www.unido.org/doc/51262; www.unido.org/en/doc/71847#

Annex C. Greenhouse Gas Footprint Policies and Initiatives Assessment

World Bank Agency-wide climate policy



The World Bank does not have a climate change policy at this time, although it does have a stated commitment to address climate change issues both as a development agency (to promote low-carbon development) and an organizational entity (to reduce and offset its footprint). A comprehensive strategic framework for World Bank engagement, including support for developing countries’ efforts to adapt to climate change and achieve low-carbon energy growth, is in the consultation phase within the World Bank and is expected to be presented at the 2008 annual meetings for endorsement by the Board of Governors.

Energy efficiency program



The World Bank has worked to increase its building energy efficiency through many measures including building operational efficiency, equipment changes, and engineering attention. Its decentralization goal has decreased the number of staff working in its Washington offices. This, combined with technology to allow long-distance teleconferencing, has reduced unnecessary business travel.

Annual GHG inventory



The World Bank completed a 2006 GHG inventory of its U.S. operations for EPA’s Climate Leaders program and is beginning to prepare a worldwide 2007 GHG inventory which will become an annual exercise.

Reduction targets and metrics



As part of its EPA Climate Leaders participation, the World Bank pledges to reduce total U.S. GHG emissions by 7 percent from 2006 to 2011. The International Finance Corporation, a World Bank agency, is not part of this commitment. In FYs 2005, 2006, and 2007, the World Bank met a carbon neutral target. This includes emissions associated with operating its Washington-area buildings, business flights tracked from Washington, as well as transportation and hotel stays for spring and annual meetings for delegates and staff. The World Bank has purchased renewable energy certificates to cover 100 percent of the electricity used in its Washington offices. These certificates pay for the incremental cost of using clean energy instead of traditional fossil fuel energy. To offset the remaining carbon emissions in FY 2007, it purchased 59,400 metric tons of carbon dioxide equivalents in verified emissions reductions from projects in Moldova and Costa Rica.

Governing strategy

Employee engagement program

No information was available on the World Bank’s integrating climate change into its core operations, although mainstreaming environment is a key plank of the Bank’s Environment Strategy published in 2002. 

The World Bank Group subsidies to employees for the use of public transportation.

Centralized GHG data management process

No information could be found about the World Bank’s GHG data management process.

Public disclosure

The World Bank discloses information about its external climate change program to the public. No information on its internal climate activities is available.

Initiatives to assess and mitigate impacts of loans, projects, and investments

The World Bank administers nine carbon funds and facilities through the Carbon Finance Unit, leveraging public and private investment for projects to reduce GHG emissions. In FY 2006, 27 Emission Reductions Purchase Agreements were signed for a total volume of 148 million tons of carbon dioxide equivalent, including two Trifluoromethare (HFC-23) projects totaling 129 million tons of carbon dioxide equivalent. In the last seven years, the Bank’s carbon finance activities have grown from the pioneering $180-million Prototype Carbon Fund to funds and facilities that represent almost $1.6 billion, with more than 62 private sector companies and 13 governments as participants. The World Bank is also working on a policy for assessing the climate risk of their loans, projects, and investments.

Information sources

http://go.worldbank.org/75DLOCZFG0; http://go.worldbank.org/W13H8ZXSD1; http:// go.worldbank.org/VRETHAGHE0; discussions with World Bank Group environmental staff

GEF Annual Performance Report 2007

Annex D. Methodological Notes on the Performance Matrix This annex briefly describes the considerations taken into account for each of the performance matrix’s 13 parameters.

D.1 Project Outcomes Although the figures on project outcomes are projected to be four-year moving averages based on the terminal evaluation reports submitted in the preceding years, including the fiscal year for which the APR is being presented, the figures presented in this year’s APR are based on the terminal evaluation reports submitted during FYs 2005, 2006, and 2007. This is because project outcomes were not assessed for the FY 2004 cohort. The aggregate figures are weighted averages, with each project considered to have equal weight.

D.2 Quality of Supervision and Adaptive Management The figures presented on quality of supervision and adaptive management are based on the findings of the pilot assessment of project supervision presented in the FY 2006 APR. The projects considered for this assessment were under implementation during FY 2005 and/or FY 2006. The figures will be updated every two to three years through follow-up assessments.

months required to prepare projects. The data on this parameter will be provided by the Agencies and the GEF Secretariat database. These figures will be updated biennially. This year, no figures are provided for this parameter yet.

D.4 Project Implementation Completion Delays The information presented in the terminal evaluation reports is the primary source for this parameter. The figures for implementation completion delays are projected to be four-year averages and are based on the information provided in the terminal evaluation reports. However, in this year’s APR, the figures provided are based on a threeyear period (terminal evaluation reports submitted during FYs 2005, 2006, and 2007).

D.5 Materialization of Cofinancing The figures for materialization of cofinancing pertain to projects whose terminal evaluation reports were submitted to the Office during FYs 2004, 2005, 2006, and 2007. The analysis is based on the information provided by the Agencies in the terminal evaluation reports or through other communications. These figures have not been verified.

D.3 Project Preparation Elapsed Time

D.6 Independence of Agency Evaluation Units

The figures presented for project preparation elapsed time will indicate average number of

This year, the GEF Evaluation Office has begun a consultation process with the evaluation units of 71

the GEF Agencies to define an appropriate way forward in assessing their independence. Broadly, the assessment provided on this parameter will be based on self-reporting by the Agencies and peer reviews carried out in the context of the Evaluation Cooperation Group of the Banks and the United Nations Evaluation Group. The charter and mandate of the various evaluation units will also provide evidence of their degree of independence. No ratings are provided on this parameter this year.

D.7 Independence of Terminal Evaluations Independence of terminal evaluations will be appraised through the assessment of the process followed for conducting terminal evaluations through field verification and will be based on interviews with relevant staff and consultants of the partner Agencies. This will allow the Office to assess the extent to which systems in the partner Agencies are conducive to unbiased and candid terminal evaluations. The following dimensions will be assessed to provide ratings on this parameter: zz

Extent to which the drafting of the terms of reference is independent of the project management team

Extent to which the recruitment of the evaluator was independent of the project management team

Extent to which the Agency recruited the appropriate evaluator for the project

Extent to which the evaluator had adequate resources (budget and time) to carry out the evaluation

Extent to which the M&E system provides access to timely and reliable information

Extent to which there was any undue pressure from management on the evaluators regarding the evaluation process (for example, in terms of site selection, selection of informants, confidentiality during interviews, information disclosure, and ratings)

Extent to which the evaluation was subjected to an independent review process

Ratings will be provided on this parameter when field verifications become a regular feature of the APR.

D.8 Realism of Risk Assessment The figures for realism of risk assessment are based on the findings of the pilot assessment of project supervision for candor and realism of supervision reporting presented in the 2006 APR. The projects considered for this assessment were under implementation during FY 2005 and/or FY 2006. The figures will be updated after a two- to threeyear period through follow-up assessments.

D.9 Robustness of Program Result Indicators and Tracking Tools The assessment of robustness of program result indicators and tracking tools will remain unreported in the 2007 APR. Given the highly specialized and technical nature of this assessment, the Evaluation Office will take up this exercise as part of the program studies for the OPS4 in FY 2008, enlisting the assistance of the appropriate technical experts.

D.10 Quality Assurance of Project M&E Arrangements at Entry The assessment of quality assurance of project M&E arrangements at entry was carried out in the 2005 APR and was based on a review of the M&E

GEF Annual Performance Report 2007

plans of the project appraisal documents that were endorsed by the GEF Chief Executive Officer in that fiscal year. The Evaluation Office will update the ratings on this parameter every two to three years through follow-up assessments.

D.11 Quality of Project M&E during Implementation Figures on quality of project M&E during implementation are based on review of the terminal evaluation reports submitted to the Evaluation Office. The figures need to be four-year running averages of the percentage of projects rated moderately satisfactory or above in M&E during implementation. However, for this APR, the figures reported in the matrix are a weighted average, with each project having an equal weight, of the data from the review of the reports submitted during FYs 2006 and 2007. In due course, as data for subsequent cohorts become available, the figures presented will shift to a four-year running average.

D.12 Quality of Project Terminal Evaluation Figures on quality of terminal evaluation reports are based on the ratings provided by the Evaluation Office after their review. For this parameter, twoyear running averages are used, with each project having an equal weight. The figures presented in the matrix pertain to FYs 2006 and 2007.

Annex D. Methodological Notes on the Performance Matrix

D.13 Quality of Learning: Improvement in Performance The performance matrix will also report on an assessment of the improvement demonstrated by GEF Agencies and entities on the other 12 parameters included in the performance matrix. This section of the matrix will be accompanied by a narrative that explains the areas in which learning has taken place and will identify the specific changes or factors that have contributed to improved performance. Ratings on improvement in performance on individual parameters will be provided using the following scale: zz

4—significantly improved or maintained excellent performance

3—marginally improved or maintained good performance

2—marginally deteriorated or maintained mediocre performance

1—significantly deteriorated or maintained poor performance

In this performance matrix, only the changes in quality of terminal evaluations (parameter 12) have been reported on. A two-year running average of quality ratings for the terminal evaluation reports submitted during FYs 2006 and 2007 has been compared with the baseline for the FYs 2004 and 2005 cohort.

Annex E. Management Response This annex presents the management response to the executive version of this report, which was presented to the GEF Council in April 2008 as GEF/ ME/C.33/2. Minor editorial corrections have been made. Note that the paragraph numbers cited refer to the executive version of the report. The Secretariat welcomes the GEF Annual Performance Report 2007, prepared by the GEF Evaluation Office. The 2007 APR provides an assessment of project outcome ratings, materialization of cofinancing, quality of monitoring and evaluation, and delays in project completion. This year’s APR also for the first time presents a performance matrix summarizing the performance of GEF Agencies on a number of parameters, some of which are still being developed. This year’s APR furthermore focuses on capacity development projects and on the GHG emissions of the Agencies’ internal operations. The Secretariat welcomes the report’s conclusion that completed GEF projects remain on target to achieve 75 percent satisfactory outcomes, a target that was agreed upon in the GEF-4 replenishment agreement. The report notes that the number of terminal evaluation reports submitted in FY 2007 (42) was lower than the number of reports (66) submitted in FY 2006, and the Evaluation Office states its intent to investigate this difference. The Secretariat notes that the number of reports submitted in FY 2005 (41) was almost the same as for FY 2007. 74

The Secretariat notes with some concern the finding of only 75 percent materialization of the promised cofinance for the cohort of projects that submitted terminal evaluation reports in FY 2007. Although the degree to which cofinancing actually materialized has varied from year to year, and the average of the last four years is still above 90 percent, the Secretariat finds that this issue deserves special attention in the 2008 APR. The Secretariat also notes the finding of an average delay of project completion of about one year. The delays seem to be decreasing, and the Secretariat would like to see this possible trend further analyzed in the 2008 APR. The Secretariat welcomes the finding on the positive results of GEF capacity development activities and notes the apparent weaknesses in relation to sustainability. The work undertaken by the Evaluation Office on evaluation of capacity development is very useful to determine the best way to improve the delivery and sustainability of capacity building across projects. The capacity development strategy approved by the GEF Council is now being implemented, and the comments and suggestions received in this year’s APR will contribute to making it more effective. The Secretariat also acknowledges the comments on capacity development given in chapter [5] of the APR and notes the Evaluation Office’s views

that the GEF portfolios in the Philippines and Vietnam are generally positive and in some areas significant. It is further noted that the effectiveness has been varied in terms of results. As the evaluation reports highlight, there are limited incentives in the government to retain and reward motivated staff, and capacity development is also closely linked to institutional structures and procedures as well as progression and repetition in training rather than “one-shot” solutions. The Secretariat appreciates the insights and recommendations received. These will be taken into account in future in order to strive to improve the delivery, effectiveness, and efficiency of capacity development. The Secretariat welcomes the significant improvement of the quality of the terminal evaluation reports and notes the observed correlation between the quality at entry of M&E arrangements and the actual quality of monitoring. The Secretariat welcomes the progress of the Agencies toward minimizing the carbon footprints of their internal operations, as documented by the assessment undertaken by the GEF Evaluation Office. Of greater interest, however, will be the resulting climate change policies and strategies being prepared for adoption by the Agencies and the resulting impacts that these will have on the Agencies’ portfolios. In response to the recommendation that the GEF Secretariat, in consultation with the GEF Agencies, conduct an in-depth assessment of the GEF Agencies’ efforts to reduce GHG emissions, the GEF Secretariat finds that this task falls beyond its mandate and should not be considered a priority. As noted by the APR, the Agencies are making progress toward understanding and managing their carbon footprints and are cooperating with each other and the UN Environment Management Group in this area. This finding indicates Annex E. Management Response

that there is little or no formal role to play in this area for the Secretariat. The Secretariat welcomes the progress toward a performance matrix that summarizes the APR findings in a set of quantitative indicators. The Secretariat notes that this tool is still under development and that the scores indicated for the Secretariat so far are set as the aggregated scores of the Agencies. Additional comments to this year’s APR have been received from UNEP, UNDP, and the World Bank. UNEP notes the following: UNEP agrees with the Evaluation Office that some parameters in the Agency Performance Matrix can only be realistically assessed at 2- or 3-year intervals. This is the case for parameters 2, 8, and 10 in table [10.1]. However, given that this APR presents ratings based on the 2005 APR findings and on the quality of project supervision exercise performed in 2006, UNEP wishes to stress that such ratings do not correspond to current UNEP performance levels. For the last two years UNEP has implemented several measures to strengthen project M&E systems and improve quality of project supervision which have been previously reported to Council as part of the Management Action Record. An indicator of improved levels of performance is the 73 percent rating on quality of project M&E during implementation (table [10.1], parameter 11), which is based on FY07 terminal evaluation findings.

UNDP has noted the following points: UNDP tracks when its terminal evaluation reports [TERS] are completed and submitted to the GEF Secretariat. For example, a total of 33 TERs will be completed in UNDP’s FY 2007 (year end) and submitted to the GEF Secretariat which is a considerably larger number than the 18 UNDP TERs that the GEF Evaluation Office used to make its conclusions and recommendations in this APR for GEF FY 2007. UNDP regards the Performance Matrix as potentially useful for fair and transparent benchmarking provided the criteria for measuring each parameter are objectively defined, the parameters remain constant from

year to year, and the time periods for reporting are clear—we note for example that the current version mixes current and historical data.

The World Bank has noted the following points: On a general note, with the exception of the Institutional Performance Matrix, the report presents aggregated findings, conclusions, and recommendations across all the Agencies, with no differentiation. Our preference would be to avoid such distinctions among Agencies but it is clear from the Performance Matrix that the World Bank recorded high ratings on all but one of the indicators (quality at entry of M&E) and the highest among the Agencies in nearly all cases. Generalizing on the results especially given the implications for specific Agency action masks these differences and provides a one-sizefits-all approach. In previous years, Agencies that were particularly at fault in specific areas were identified. For example, conclusion 1 gives the overall outcome ratings in the satisfactory range as close to the 75 percent target. The World Bank’s result was 87 percent, which is significantly above the satisfactory range. Para 2. In the case of the World Bank, the GEF Evaluation Office has previously stated that it accepts the findings of IEG’s project reviews. Therefore, there is no uncertainty in the verification process. Perhaps the Evaluation Office needs to more fully involve the central evaluation units of other Agencies in the process, but it not only already works closely with IEG but has modeled its own project reviews on IEG’s processes and formats. Similarly, with regard to the reference in the paragraph to piloting a verification methodology in the field, IEG’s Project Performance Assessment Report is used for this purpose and has been previously accepted by the Evaluation Office. It would be useful, therefore, to provide a footnote indicating the status of independent evaluation in the World Bank. Para 6. Is there evidence that information on project completion is not available in a timely manner? Is there an assessment of the lag between preparation of the completion report and its receipt by the Evaluation Office? In the Bank’s case, all completion reports for FSPs are automatically sent to the Evaluation Office at the same time it is circulated to other units within the Bank. As a result there should be no lag in receipt of the reports, as well as IEG’s review.

Para 19 and 20. No evidence is presented on why there should be doubts about the sustainability of capacity building. To say that sustainability is low because few national or long-term training programs were developed focuses on a continuation of the provision of training rather than on whether the training provided by the project enabled better institutional or individual performance. Moreover, capacity development is broader than training per se. Development of longterm training programs also has long-term financial implications. Para 27. The Bank’s Implementation Completion Report template includes fields for financial reporting and for analysis. Again, the generalization of the findings probably masks different practices among the Agencies. Para 30. At the end of the paragraph, we suggest adding the following text: “though the financial costs may be higher.” It would also be useful to know whether there is a cutoff point when the costs under the direct route followed by the GEF Secretariat are so high that it outweighs the GHG benefits and therefore carbon neutrality could be more efficiently obtained from purchasing carbon offsets. Para 33. This is another example of the generalizations. There is a difference between those Agencies that achieved six or seven of the criteria and those that have only achieved one (International Fund for Agricultural Development) or two (UNDP). We earlier pointed out in written comments that the Bank does include a climate policy in its overall strategy, through the broader greening program, which for example includes energy efficiency in buildings. Performance Matrix. It is not clear why there is missing information in the case of the Bank for the following. For line 3 the Bank provided information on elapsed time as part of the Project Implementation Review. Line 6, IEG’s independence is well established. Line 7, IEG does an independent review of the terminal evaluations. The result for the Bank in Line 13 (11) conveys the wrong impression. The quality of the Bank’s terminal evaluations has consistently been rated in the 80 percent range and hence there is far less room for improvement. The rating of 3 conveys the wrong impression.

GEF Annual Performance Report 2007

References The GEF Council documents cited here (indicated with the designation “GEF/C.xx”) are available on the GEF Web site, www.thegef.org, under Documents/Council Documents. GEF Evaluation Office documents can be found on the GEF Evaluation Office Web site, www.gefeo.org, under Publications.

GEF EO (Global Environment Facility Evaluation Office). 2006. The GEF Monitoring and Evaluation Policy. Evaluation Document No. 1. Washington, DC.

GEF (Global Environment Facility). 2000. “GEF Project Cycle.” GEF/C.16/Inf.7.

OECD DAC (Organisation for Economic Co-operation and Development Development Assistance Committee). 2002. Glossary of Key Terms in Evaluation and Results Based Management. www.oecd.org/ dataoecd/29/21/2754804.pdf.

—. 2006. “Summary of Negotiations on the Fourth Replenishment of the GEF Trust Fund.” GEF/C.29/3.

—. 2007. “Guidelines for GEF Agencies in Conducting Terminal Evaluations.” Draft.

GEF Evaluation Office Publications

Number

Title

Year

Evaluation Reports 44 GEF Annual Country Portfolio Evaluation Report 2008 43 GEF Country Portfolio Evaluation: South Africa (1994–2007) 42 GEF Country Portfolio Evaluation: Madagascar (1994–2007) 41 GEF Country Portfolio Evaluation: Benin (1991–2007) 40 GEF Annual Performance Report 2007 39 Joint Evaluation of the GEF Small Grants Programme 38 GEF Annual Performance Report 2006 37 GEF Country Portfolio Evaluation: Samoa (1992–2007) 36 GEF Country Portfolio Evaluation: The Philippines (1992–2007) 35 Evaluation of the Experience of Executing Agencies under Expanded Opportunities in the GEF 34 Evaluation of Incremental Cost Assessment 33 Joint Evaluation of the GEF Activity Cycle and Modalities 32 GEF Country Portfolio Evaluation: Costa Rica (1992–2005) 31 GEF Annual Performance Report 2005 30 The Role of Local Benefits in Global Environmental Programs 29 GEF Annual Performance Report 2004 28 Evaluation of GEF Support for Biosafety Third Overall Performance Study GEF Integrated Ecosystem Management Program Study Biodiversity Program Study Climate Change Program Study International Waters Program Study

2008 2008 2008 2008 2008 2008 2008 2008 2008 2007 2007 2007 2007 2006 2006 2005 2006 2005 2005 2004 2004 2004

Evaluation Documents ED-3 Guidelines for GEF Agencies in Conducting Terminal Evaluations ED-2 GEF Evaluation Office Ethical Guidelines ED-1 The GEF Evaluation and Monitoring Policy

2008 2008 2006

Evaluation Office

G LOBAL ENVIRONMENT FACILITY

Global Environment Facility Evaluation Office 1818 H Street, NW Washington, DC 20433 USA www.gefeo.org