Evaluation of Poverty Reduction Programs. UNDP experience in Kazakhstan

EVALUATION OF NATIONAL POVERTY REDUCTION PROGRAMS UNDP experience in Kazakhstan

MASSIMILIANO RIVA (massimiliano.riva@gmail.com)

ABSTRACT This paper reviews the United Nations Development Programme (UNDP) experience in conducting national wise evaluations in Kazakhstan. Namely the evaluation of the National Program for Poverty Reduction of Kazakhstan 2003-2005, and the outcome evaluation of the UNDP Poverty Reduction Program 2000-2005 are analyzed.

UNDP has been focusing to achieve clearly stated results of its activities. As part of these efforts UNDP has shifted from traditional project monitoring and evaluation (M&E) to results-oriented M&E and outcome evaluations. An outcome evaluation assesses how and why an outcome is being achieved in a given context, and investigates single contributions to the outcome.

The main purpose of the mentioned evaluations is to assess the achievements toward poverty alleviation in Kazakhstan and to draw recommendations and lessons learned for future policies. This parallel exercise offers interesting opportunities to study different methodologies and approaches used in evaluating national development strategies and programs.

This paper then focuses on sustainability and ownership transfer and the way they can be measured by program evaluations, using the experience of UNDP in Kazakhstan as background. Finally lessons learned that may have generic application are highlighted as well as innovations and best practices.

The views expressed in this report are those of the author and do not represent the views of UNDP. This paper is a product of the author only and has not been cleared or officially transmitted to UNDP.

INDEX of CONTENT 1

Introduction

2

UNDP and sustainable development

3

Outcome mapping and result based management in UNDP

4

Outcome Evaluation of the UNDP Poverty Reduction Programme 4.1 Methodology 4.2 Performance assessment 4.3 Sustainability 4.4 Development changes and attribution challenge

5

Evaluation of the Poverty Reduction National Program 5.1 Development context 5.2 Methodology 5.3 Performance assessment 5.4 Sustainability

6

Lessons learned

7

References

2

1 Introduction The United Nations Development Programme (UNDP) has emerged as the leading UN's global development network, being an organization that advocates for change and connects countries shares knowledge, experience and resources to help people build a better life. There are 166 country offices around the world, working to develop solutions to overcome development challenges.

UNDP has been focusing to achieve clearly stated results of its activities. As part of these efforts UNDP has shifted from traditional project monitoring and evaluation (M&E) to results-oriented M&E, and outcome M&E. An outcome evaluation assesses how and why an outcome is being achieved in a given context, and investigates single contributions to the outcome. However outcomes are influenced by the activity of many development agents and actors. Outcome evaluations help to clarify underlying factors affecting developmental changes, highlight unintended consequences, recommend actions to improve future programming, and generate lessons learned.

This paper focuses on UNDP’s experience in evaluating poverty reduction programs in Kazakhstan. UNDP started its activities in Kazakhstan back to 1993, shortly after the country's independence. Since then, the office has implemented over 100 projects and assisted the government in developing a number of policy documents, including the National Poverty Reduction Program 2003-2005.

UNDP in the first half of 2006 was involved in three major evaluations: the outcome evaluation of its poverty reduction programme (2000-2005), the evaluation of the national poverty reduction program (2000-2003) and the outcome evaluation of the sustainable development Programme (2005 – 2009). The evaluations addressing the national and the UNDP poverty programs are thereby analyzed, while there will be references to the sustainable development programme evaluation.

2 UNDP and Sustainable Development UNDP has championed the concept of Human Development, defined as expanding the choices for all people in the society. This means that men and women - particularly the poor and vulnerable - are at the centre of the development process. It also means "protection of the life opportunities of future generations..." (UNDP Human Development Report, 1996). In many strategic documents the UNDP’s mission of promoting sustainable development is clearly mentioned.

An outcome evaluation is intended to address, as per UNDP Guidelines:  Identification of innovative methodologies to approach key development issues of Sustainable Human Development;  National capacities developed through UNDP assistance;  Level of participation of stakeholders in the achievement of the outcome;  Identification of direct and indirect beneficiaries and their perceptions of how they have benefited from the UNDP assistance;

3

3 Outcome Evaluations and Result Based Management in UNDP Since 1999, UNDP has pursued a reform programme to reveal how and where the organization is making measurable development results. UNDP envies results-based management (RBM) defined as a methodology in which performance is measured at the level of development goals and outcomes, and resources are strategically managed to enhance the organization’s development effectiveness -. UNDP has shifted toward the promotion of a “culture of performance”, trying to realign existing tools and to design new programming tools. The Handbook on Monitoring and Evaluating for Results was developed as the main corporate guide to monitoring and evaluation.

As stated in UNDP guiding documents M&E the main objectives of RBM monitoring & evaluation are:  Enhance organizational and development learning;  Ensure informed decision-making;  Support substantive accountability and UNDP repositioning;  Build country capacity in each of these areas, and in monitoring and evaluating functions in general.

Within the RBM framework, UNDP uses at least three types of indicators, also known as results indicators:  Situational (impact) indicators provide a broad picture of whether selected developmental changes are occurring;  Outcome indicators assess progress against specified outcomes;  Output indicators assess progress against specific operational activities.

Figure 1

Assessing performance along results DEVELOPMENT PROBLEM / ISSUE

Situational Indicators

Senior Management

INPUTS ACTIVITIES

Output Indicators

Project Management

Outcome Indicators

Programme Management

OUTPUTS

OUTCOMES

SUSTAINED IMPACT

Long-term Impact Indicators

Senior Management

Source: RBM in UNDP: Selecting Indicators

4

The periodical evaluations follow the result chain scenario where input and outputs affect outcomes and produce long-term impacts (Figure 1 and 2).

Figure 2

Result chain

INPUTS

OUTPUTS

OUTCOMES

IMPACT

• Experts

• Studies completed

• Income increased

• Health conditions

• Equipment

• People trained

• Jobs created

• Funds

improved • Longevity increased

Source: UNDP Handbook on Monitoring and Evaluating for Results

The introduction of outcome evaluations is fairly new in UNDP, being implemented only from 2002. Indeed lessons learned still have to be discussed and shared among country offices. The existence of knowledge networks and the work of the Evaluation Office in the Headquarters are now facilitating this experience sharing. The findings of this paper cannot be generalised for UNDP, but can open space for reflection and improvement.

UNDP work is becoming every day more inter-linked with the operations of UN sister agencies.

Most of the countries have developed the so called United Nations Development Assistance Framework (UNDAF), where UN outcomes are agreed after discussions between UN agencies, the Government and other stakeholders. Different UN organizations may work for the achievement of the same outcome, even if this is not the case thereby studied in Kazakhstan.

UNDP Kazakhstan has few strategic documents in which outcomes and indicators are defined:  United Nations Development Assistance Framework (UNDAF) 2005-2009 represents the strategic framework for the country-level activities of the entire UN system  Country Program Document (CPD) 2005-2009 sets out the cooperation framework between the Government of Kazakhstan and UNDP and defines the expected results from UNDP  Country Program Action Plan (CPAP) 2005-2009 sets out the programmatic workplan of UNDP Before 2005 different labels were used in Kazakhstan, namely Country Cooperation Framework (CCF) 2000-2004 and Strategic Result Framework (SRF) 2000-2003. The CCF and the UNDAF and SRF and CPD are respectively documents with similar structures and rationality. The different acronyms are due to the evolution/reform of the UN system and are linked with an increased significance of coordination among UN agencies and joint UN projects.

4 Outcome Evaluation of the UNDP Poverty Reduction Programme 4.1

Methodology

The overall objective of the outcome evaluation of the UNDP Poverty Reduction Programme 2000-2005 is to evaluate the progresses toward outcomes as well as to draw lessons learned and practical recommendations for future planning.

5

The substructure of the evaluation is based on a set of questions grouped as below:  Status of the outcome (e.g. have the outcomes been achieved or has progress been made towards the achievement?)  Factors affecting the outcome (e.g. how have these factors limited or facilitated progress towards the outcomes?)  UNDP contributions to the outcome through outputs (e.g. were UNDP’s contributions to the achievement of the outcomes appropriate, sufficient, effective and sustainable?)  UNDP partnership and resource mobilization strategy (e.g. were partners, stakeholders and/or beneficiaries of UNDP assistance involved in the design of UNDP interventions in the outcomes area?)  Lessons learned (e.g. what are the best and worst practices in designing, undertaking, monitoring and evaluating outputs, activities and partnerships around the outcomes?)  Recommendations (e.g. what corrective actions are recommended for the new, ongoing or future UNDP work in the CPAP outcome?)

The necessary synthetic approach focused on project clusters, leaving behind project by project in-depth evaluations. What the methodology clearly states is the shift from project results to aggregated results of UNDP interventions.

The evaluation focuses on UNDP’s contribution to poverty alleviation during two different programmes cycles. The evaluation was conducted at an early stage of the 2005-2009 cycle and 2 years after the completion of the 2000-2004 cycle.

In the framework of UNDAF Outcome 1 "Reduced (income and human) poverty at national and sub-national level” the following outcomes were formulated:  SRF Outcome (2000-2004): formulation of poverty eradication and employment promotion programmes at central and local level / National Anti-Poverty Programme Indicator: increase in the number of laws and amendments developed by Parliament Baseline: one draft law in environmental protection  CPAP Outcome (2005-2009): The Government is more capable of reducing poverty, achieving MDGs targets, and monitoring its progress to these ends Indicator: increase in the allocation from national budget to poverty alleviation initiatives Baseline: in 2002 1.9% of GDP health care, 3.2% education and 5.6 % social services In principle UNDP guidelines are very basic: evaluators have to rate changes toward outcomes as: a) positive change b) negative change, or c) unchanged.

A descriptive analysis substantiates the rating. Evaluators used the so-called method of triangulation, where perceptions formed through interviews and qualitative analysis are validated through quantitative data, statistics and tracking of indicators, and documented by reviewing project and programme reports.

6

4.2

Performance assessment

Following the methodology presented in the previous chapter, the performance of the organization has been assessed through:  the assessment of changes toward the outcome (indirectly)  the analysis of UNDP contributions toward the outcome  the evaluation of UNDP partnership strategy  the evaluation of UNDP resource mobilization strategy  the best and worst practices  the assessment of sustainability of UNDP interventions  the assessment of ownership transfer to relevant stakeholders The starting point was the two outcome indicators specified in the CPAP/SRF documents:  Increase in the number of laws and amendments developed by Parliament  Allocation from national budget to poverty alleviation initiatives The UNDP office was conscious of the inadequateness of the above outcome indicators and the ToR required the development of additional indicators. Therefore the evaluators developed the following:  Number of poverty eradication and employment promotion programmes at central and local levels approved;  Needs assessment undertaken and action plans on human security developed in the Aral Sea, Caspian Sea and Semipalatinsk regions;  The number of regions where micro-finance programmes have been replicated;  New improved methodologies for targeting social assistance approved and being applied;  Greater awareness of poverty problems (that will be assessed in relation to the impact of with the UNDP reports on national debate and policy changes);  Legislation and regulation on social sector NGOs developed.

The development of additional indicators helped to substantiate the assessment over the progresses toward outcomes. Apart from the above indicators the evaluation drew a link between UNDP outputs and outcomes by assessing the influence of UNDP interventions, i.e. the influence of specific projects and soft assistance on policy formulation and implementation, trying to discern what institutional and policy changes UNDP assistance brought about.

The assessment of outcomes often risks to results sensitive and subjective, since many players and externalities (positive and negative) could have contributed as much (or more) than UNDP. Therefore the analysis of underlining factors affecting the outcome helps to disaggregate the causes of changes and to ground different contributions (e.g. USAID microfinance projects) and externalities (e.g. high oil prices).

The analysis started with the attempt to interlink project outputs to progresses toward outcomes.

With that methodology evaluators disaggregate UNDP intervention in project outputs (or deliverables) and soft assistance. Soft assistance indeed cannot be traced looking at expected outputs and workplan; it takes the form of policy advice, policy dialogue, and advocacy, brokerage/coordination provided by UNDP outside of established projects or programmes.

7

The shift form the rating of development changes to projects outputs moves the attention inside the organization. However, the analysis of project output was not an easy task and different challenges emerged during the evaluation of the UNDP Sustainable Development Programme (environment and energy practice projects). The UNDP poverty programme covered 26 projects that were implemented in different geographical regions with different life spans often. To ease the evaluation 3 projects clusters has been created:  Pro-poor policies and poverty monitoring  Private sector development  Quality of life in Semipalatinsk The Poverty Programme is itself a big cluster of projects that belong to the world practice “Poverty Reduction”. The implementation of projects is highly diverse as well as the characteristics of the implementing agency; i.e. projects were implemented by the Government,

by NGOs and by ad hoc UNDP project structures. The UNDP involvement in the project management and the degree of supervision varied on the basis of the priority of the deliverables and the capacity of the implementing partners (or project management).

Only recently the use of modern result based management (RBM) tools has emerged in UNDP and, indeed, the Kazakhstan office lacks an independent Monitoring and Evaluation (M&E) unit.

The headquarter pressure on RBM and evaluation challenges country offices scarce human and financial resources. The M&E functional responsibility is usually conferred to programme core staff, while, on ad hoc basis, independent consultants are hired to conduct project or outcome evaluations. Among the 26 poverty practice projects under evaluation, only 3 projects were evaluated.

The UNDP reporting system is relatively “light” if compared to other international organizations.

Yearly targets and indicators are set at the programme level (e.g. poverty reduction practice) not at the projects level - and constitute the internal monitoring system. The external evaluations conducted in 2006 are also part of the UNDP corporate evaluation requirements. However, the focus on outcomes and service lines has been sometimes detrimental to project monitoring, which is often made only by descriptive annual progress reports. There is a general lack of clear indicators and targets (except for identification of outputs or deliverables) set at the project level.

Consequently the assessment of projects effectiveness was difficult due to the time constraints of an outcome evaluation and to the inadequate project monitoring and evaluation.

No traditional organizational performance indicators were developed in the organization. To collect relevant information, consultants were asked to comment on UNDP KZ partnership strategy and resource mobilization strategy (key aspects of UNDP interventions). The main source of information was meeting/interviews with UNDP partners and “competitors”. The position of UNDP was also compared with other donors. Best practices were then highlighted.

UNDP co-operation/partnership strategy was studied and grouped around main partners:  UN agencies  Government  International donors  NGOs

8

The relationship has been evaluated focusing studying:  The implementation of UNDP projects (Government, NGOs)  Joint programming (UN)  Sharing of relevant information (all)  Coordination of interventions (all)  Beneficiaries feedbacks The resource mobilization strategy was similarly analyzed. Evaluators looked at UNDP achievements and to its potential to attract additional funding and to potential sources of funding (e.g. international donors, the private sector and the Government of Kazakhstan). Definitely the office was in a transition stage, facing the challenge of shifting from traditional donors toward hosting government funding (or co-finance agreements). The previous experiences with international donors were reviewed, but opportunities with the private sector and the government received more attention since were central for the future of the organization. Also in this case meeting/interviews with previous and potential donors were the main information sources.

In addition, management structures and working methods were evaluated in order to assess the appropriateness and effectiveness of UNDP work. The organizational analysis, even if qualitative and descriptive, resulted to be sharp and well-grounded as well as the recommendations. The appropriateness of UNDP intervention at large was at the center of the second part of the evaluation and the un-structured approach brought flexibility and gave the opportunity to pick up the most relevant organizational challenges. Even if this approach may seem less holistic and structured than usual organizational performance assessments, it resulted to be effective in addressing staff concerns and in providing alternative options to the management. Cost effectiveness and quality of UNDP intervention and human resources adequateness were evaluated as well as predominant working practices (coordination, internal learning, information sharing, institutional memory, etc.).

UNDP comparative advantages were also highlighted. During the evaluators’ mission in Kazakhstan, meetings with donor and government agencies were used to test the UNDP ideal positioning in the area of poverty reduction.

As mentioned in the introductory chapters, the concepts of sustainability and sustainable development are well integrated in UNDP working practice. Every project has a defined exit strategy. However, sustainability concerns are stronger at the project level than at the outcome level. In a more generalized way UNDP intervention as whole was evaluated to assess its appropriateness toward long term goals. As a result, without proper established indicators at the outcome level, and with the impossibility of project by project in-depth analysis, the analysis of sustainability was not detailed. In addition the timing of this evaluation (e.g. the conduction during the first year of the new programme cycle) did not allow to properly test the sustainability of progresses toward the outcome.

The report highlighted different results in terms of sustainability and replication effects among the projects. The yearly progress reports have to report on sustainability, while more developed indicators are built for specific components (e.g. microfinance or business incubators).

The national ownership concept is also central for UNDP interventions, other than being itself a driver of sustainable development. National ownership is defined as the degree of genuine host 9

country commitment to development initiatives and polices (e.g. political directives and statements, allocation of government resources, etc.); the strength of public participation, and, the involvement of local institutions in planning, implementation and evaluation are some of its most prominent attributes. Even if widely acknowledged as one of the most relevant development factors, national ownership is rarely properly captured in evaluations. In the final report national ownership was studied, even if the assessment was not structured and diluted in the text. The evaluators looked at:  Links between national priorities and UNDP interventions  National execution of UNDP projects  Partnership strategy with local institutions  Government support (financial and non-financial)  Use of national experts and national implementing partners 4.3

Development changes and attribution challenge

In UNDP literature an outcome is defined as an intended change in development conditions between completion of outputs and achievement of impacts. The broad outcomes often relate to reduction of poverty, improving access to safe water, arresting the spread of HIV/AIDS, etc. It is then well understood that the challenge for the development community is to clearly define and monitor progresses toward outcomes. On the other side the demand from both citizens of donor and recipient countries has increasingly raise the pressure on UNDP to clearly show the results of its work.

Per se outcomes present significant challenges in terms of measurement, and even more subtly, of attribution. Indeed clear indicators, targets, and baselines are needed to demonstrate organizational contributions. Then a clear understanding of partnerships, areas of influence, and comparative advantages is necessary.

Organizations have indeed to define their sphere of influence. In view of achieving national and global development targets (e.g. MDGs), shared responsibility and accountability are primary concerns. Then the question is “how” development actors conduct and monitor their business.

The methodology used by UNDP follows a structured tree that prioritizes the achievement of the outcome (e.g. the government is better able to alleviate poverty), and then highlights the factors (positive and negative) that influence progresses toward the outcomes. Finally the analysis looks at outputs, activities, and soft assistance and their contribution to the outcome. However there are no clear guidelines on how to interlink outputs with outcomes and on how to attribute the responsibility of changes. Indicators, targets and baselines do not help since are often measures of progresses toward outcomes and provide only a weak correlation with the organization (e.g. number of pro-poor laws approved). The evaluation then tends to be based only on qualitative and often subjective analysis. One priority is to identify ways to collect empirical evidence to link project assessments with national development targets, so that future polices and programs can benefit from lessons learned and organizations can focus on their comparative advantages.

Almost all development organizations face the attribution problem and different methodologies have been developed to face this challenge. The shift from attribution to contribution, moving away from a top-down approach, is a starting point. Whatever is achieved by one development actor is just a contribution to development outcomes. However, a consensus on the best approach to deal with the attribution challenge has yet not been developed. Indeed it is acknowledged that 10

national outcomes are not achievable without partners. Partnerships, with donors, local institutions and the society add the “challenge of inclusion”. Therefore trial joint evaluations can be conducted, as well as new methodologies tested to be increase available knowledge. For instance the outcome mapping methodology focuses on changes in direct partners’ behavior.

5 Evaluation of the National Poverty Reduction Program 5.1

Development context

Kazakhstan has made remarkable achievements in the course of the transition from a centrally planned economy into a market economy. Extensive reforms and the country mineral wealth helped Kazakhstan to recover from the transition period which followed the country’s independence in 1991. Kazakhstan has a relatively open economy with good growth prospects and its internal and external stability is an exception in the region. After the recession of the 1990s, when the GDP fell by 39%, the economic growth picked up exceeding 9 % per year in 2000-2005. In 2005 Kazakhstan’s per capita GDP reached USD 3,686.

Despite strong economic growth Kazakhstan suffered the consequences of the 1990s recession and the disruption of the old social system inherited from the USSR, which exacerbated the country’s development needs. Unemployment and low wages became the main causes of poverty and the headcount poverty ratio reached 36 % of the population in the end of 1990s. Severe under-funding led to a deterioration of physical infrastructures of educational and healthcare providers, loss of skilled personnel and, as a result, a sharp decline in availability and quality of services. Recognizing critical needs form one side and the willingness to create a sustainable social system on the other, the Government introduced a series of balanced reforms in pensions, education, health care, and social protection.

The sustained economic growth helped Kazakhstan achieving the First Millennium Development Goal, as income poverty was reduced by half in 2004 (16%) compared to 1996 (35%). By 2004, 16.1 % of the population lived below the subsistence minimum of KZT 5,427 per month (about USD 3.5 per day - PPP). However progress towards achievement of other MDGs related to improvement in health and environmental conditions is more problematic and there are concerns that the 2015 targets will not be reached.

The first program on combating poverty and unemployment was developed for 2000-2002, while in 2002 the government started developing a second 3 years program. The Poverty Reduction Program of the Republic of Kazakhstan for 2003-2005 (later program) was approved by Decree in 2003. The program’s specificity was that for the first time in the country poverty was defined as a multidimensional phenomenon.

The national program has been successfully completed in 2005. A considerable reduction of income poverty has been achieved in the course of the programme implementation, although many challenges remain, especially with regard to combating various aspects of human poverty.

Still, a relevant proportion of the population lives with per capita incomes only slightly exceeding the subsistence minimum, remaining vulnerable to economic shocks and at risks of returning into poverty.

A new draft program, the “Territorial Development Strategy”, focuses on the creation of growth poles and country competitiveness. This strategy necessarily needs to be underpinned by sound 11

policy measures ensuring adequate investments in human capital essential for augmenting productivity and competitiveness. The Government is currently considering the need of a new dedicated program against poverty.

The challenge for Kazakhstan will be to reduce social and geographic inequities; improve the delivery of public services; protect vulnerable groups; improve government efficiency, and promote broad-based participation in the political and economic life. To ensure that the benefits of economic growth are more equitably distributed, the Government will need to improve public investments and promote economic diversification beyond the oil, gas and mining sectors.

The Government is committed to wide governance reforms, to promote effectiveness, transparency and e-government. However, a modern evaluation culture has still to emerge.

Reporting and monitoring are often seen as mandatory administrative requirements or punitive tools. The Ministry of Economy and Budget Planning (MEBP) is only now developing evaluation units and, in this transitional framework, the Ministry, proposed UNDP to conduct an evaluation of the 2003-2005 Poverty Reduction Program.

5.2

Methodology

The Programme formulated its main goal as reduction of poverty by means of economic and social policies conducive to the improvement of living standards. It was indeed the first comprehensive programme addressing poverty from a human development perspective. The program implementation was planned and coordinated with existing different state and sectoral programmes (e.g. further deepening of the social reforms, agricultural and food program and other sectoral programmes on health, education, social protection, demographic development). In addition the programme was decentralized in its implementation with region administrations responsible for regional poverty programs.

UNDP was contracted by the MEBP to conduct an independent evaluation of the program in the first half of 2006. The client was indeed the Government (MEBP) and the purpose of the evaluation was both an assessment of the program’s implementation as well as the development of a set of recommendation for future policies. The ToR of the evaluation and the methodology were discussed with the Government and other stakeholders (e.g. NGOs, donors).

This evaluation was an additional instrument to the monitoring reports prepared under the responsibility of the MEBP during and after the program implementation. The MEBP provided quarterly monitoring reports and a final report.

The substructure of the evaluation is based on a set of questions grouped as below:  Status of poverty – Program objective (e.g. has the poverty been reduced and has progress been made towards better living standards of the poor?)  Factors affecting poverty reduction (e.g. what factors - political, sociological, economic, etc. - have affected poverty alleviation, either positively or negatively?)  Program contributions to poverty reduction (e.g. were Program contributions to reducing poverty appropriate, sufficient, effective and sustainable?)  Lessons learned (e.g. what are the main lessons that can be drawn from the Program experience?)

12



Recommendations for future interventions (e.g. based on the above analysis, how should the Government adjust its policies, partnership arrangements, and management structures to be more effective in reducing poverty?)

The evaluation process has been divided in 2 phases:  Preparatory phase (collection of data and background studies)  Evaluation phase (data analysis and evaluation) Preparatory phase. The preparatory phase included the preparation of background studies, the conduction of field surveys and FGD and the collection of statistical data, namely:  Brief report on Program indicators  Evaluation of the Kyzylorda Poverty Program (FGD, and semi-structured interviews) and Household survey in Kyzylorda (follow up to 2003 survey)  Evaluation of the East Kazakhstan Poverty Program (FGD, and semi-structured interviews)  Desk review of relevant publications and collection of statistical data Evaluation phase. During a 3 weeks mission in Kazakhstan more than 32 stakeholders were interviewed. Evaluators used both secondary information available from government statistics, and relevant literature and primary information gathered from interviews, field surveys and FGD to draft the report.

The contents included a summary of Program objectives; an assessment of the Program strengths and weaknesses, an assessment of the achievement toward targets; a review of factors that influenced Program outputs and the achievement of the outcome; a summary of lessons learned, and recommendations for future pro-poor policies.

The evaluation assessed progress against the main objective of the Program: "Poverty reduction through implementation of complex measures for the improvement of basic economic and social factors influencing the living standards of the population”. The progress against the 15 performance indicators was reviewed. The conclusions on the status of the outcome are based on updated poverty profiles for groups at poverty risk and factors of vulnerability. It is important to note that a participatory approach was adopted, giving the program’s beneficiaries an opportunity to comment on Program’s results (e.g. public works).

While analyzing the factors contributing to poverty reduction a particular attention was paid to distinguish the contribution of pro-poor policies from the effects generated by the inflow of oil and gas revenues. In this context two relevant questions were formulated:  whether the programme itself was an effective policy instrument  where policy measures were fulfilled and objectives achieved The evaluators’ conclusions on the Program’s contribution to poverty reduction required the plausibility of associating programme outputs (actions) and outcomes based on the analysis of various primary and secondary information, especially the results of field surveys in Kyzylorda and East Kazakhstan. The evaluators did not go into details of specific interventions/activities, focusing instead on common issues and themes with a view of discerning a fresh perspective uncluttered by a detailed knowledge of projects and programmes.

13

5.3

Performance assessment

The performance assessment faced tremendous challenges due to the peculiar characteristic of the Program and the lack of a singular implementing agency. Indeed the Program’s Action Plan defined tasks for Sectoral Programs managed by line ministries (namely Ministry of Health, Ministry of Labour and Social Protection, Ministry of Education, Ministry of Agriculture and Ministry of Environment). In addition the Program was decentralized at the regional level with Poverty Reduction Programs developed by local administrations. To partially overcome scarce information two independent regional program assessments were conducted, while the evaluation focused on the coordination role of the MEBP. The performance assessment used a standard approach that highlighted:  Program strengths  Program weaknesses  Main implementing issues  The evaluators focused firstly on program performance indicators. Therefore scope, targets, appropriateness of the indicators were scrutinized. Then the 15 indicators were divided between achieved and not-achieved and the Program achievements toward targets were analyzed in details. Since some targets were not clearly related to program activities (e.g. growth of GDP per capita), the program’s contribution was assessed using additional information. The original indicators were not properly designed to assess organizational performances. When additional data was available, supplementary indicators were developed, but only an indirect assessment of organizational performance was possible.

In particular the evaluation team assessed through interviews, FGD, round-tables and available program documents and monitoring reports:  Participation of stakeholders (NGOs, etc) in the programme design, implementation and monitoring and evaluation;  Vertical (national and regional administration) and horizontal (line ministries) coordination among government agencies and the role of the MEBP;  Appropriateness of the prioritization of objectives/tasks/activities;  M&E plan, appropriateness of indicators and targets;  Appropriateness of the juridical and financial status of the program;  Dissemination of information and information exchange;  Microfinance and SME promotion funded projects.

The voice of stakeholders and beneficiaries was recorded in order to provide a comprehensive picture and to guarantee a participatory approach. The program was evaluated assessing the perceptions of beneficiaries, using interviews and focus groups conducted in the capital and in four regions.

The assessments of regional poverty reduction programs (East Kazakhstan and Kyzylorda) were essential in providing direct information and in collecting stakeholders and beneficiaries perspectives. The research agency hired to conduct semi structured interviews, FGD, and collect statistical data prepared 2 reports. The structure of those reports is similar to the one of the whole evaluation. However, the level of details is much higher. Program compliance was assessed from objectives to priorities to tasks to activities. The authors summarized also the assessment of the 14

program made by beneficiaries and the public officials for each priority (2nd level of program aggregation). Finally the comparison of stakeholders/beneficiaries perceptions and case studies enriched the findings.

Specific attention has been dedicated to gender and indeed gender mainstreaming was object of in-depth analysis. The gender analysis used available statistics and newly developed indicators, since the evaluation of a gender component was not originally planned, even if gender promotion activities were implemented.

Evaluators were asked also to highlight best practices and innovative approaches to poverty alleviation and sustainable development.

5.4

Sustainability

In this evaluation sustainability meant the sustainability of the achievements toward poverty alleviation obtained thanks to the Program implementation. Therefore a particular attention was paid to distinguish the contribution of pro-poor policies from the effects of oil and gas revenues, which gave an extraordinary opportunity to the country to recovery from the 1990s crisis.

International experiences from oil rich countries and country in transition were used in reviewing the country development context. Macroeconomic and fiscal policies were reviewed with a poverty reduction eye. The recommendations were decisively forward looking, and, as agreed with the Government, practical policy options for the future were provided.

An assessment of sustainability of poverty reduction measures against possible economic shocks, and indications on the introduction of new policies to reduce the population vulnerability were provided in recommendations.

Due to the impossibility of evaluating singular components of the program and due to the lack of objective information it was not possible to assess the sustainability of specific initiatives (e.g. public works). Microcredit and SME support projects were exceptions: financial and operational self-sufficiency (meaning permanent access to financial services) are key words in microfinance and were considered when reliable reports were provided.

6 Lessons Learned Even if lessons learned are strongly linked with the specific context of Kazakhstan, some generalization is possible: 

 

The selection of the evaluation team members is essential. The evaluation team must be balanced, trying to march the need of previous knowledge of the organization to be evaluated, independence, sectoral knowledge, language knowledge, and country knowledge; Clear definition of the evaluation objectives and priorities is a prerequisite; Clear ToRs are important assets and a prerequisite for effective discussions and roundtables with stakeholders and clients;

15

        

Clarification and in-depth discussions between evaluators, clients, and UNDP helped both the conduction of the evaluation and favored an effective use of the report afterward; The participatory approach, with adequate cover of beneficiaries is needed to ground findings and enhance an effective use of the evaluation; There is a need to address specifically gender issues in program evaluations, which is often under-represented in regular monitoring activities; Evaluations can be used as effective policy strategic documents to promote change and to improve transparency in partnership with hosting Governments; The limit of measurement. The impossibility of control (e.g. with-without trial) over a program with a sphere of influence of the one evaluated (national strategy) has privileged a qualitative evaluation focused on strengths and weaknesses; Open the judgment to the wider scrutiny. To overcome informational challenges preliminary draft of the evaluation was exposed to stakeholders for a peer review.

The conduction of the 2 evaluations involved many local researcher as well as government officials. This exercise was a capacity building tool for the government, and national research agencies and consultants; Lack of sustainability and transfer of ownership systematic evaluation. There is a need to better structure and ground the analysis of sustainability an ownership transfer; Participation (from implementation to M&E) helps to ensure sustainability in national programs and reduce the probability of “wrong” policies; increased participation of clients (in this case the Government) helped indeed to enhance the impact of the evaluation and to promote a “modern evaluation culture”.

7 References Relevant documents are available at www.undp.org or www.undp.kz UNDP. (2002) Guidelines for Outcome Evaluators, Monitoring and Evaluation Companion Series #1, UNDP Evaluation Office, New York UNDP. (2002) Handbook on Monitoring and Evaluation for results, UNDP Evaluation Office, New York UNDP. (2001) Linking Organizational and Development Effectiveness, paper presented at the joint UNDP-DFID Workshop Oxford University, 25-26 September 2001 UNDP. (2002) RBM in UNDP: Selecting Indicators, UNDP Evaluation Office, New York UNDP Kazakhstan. (2006) Evaluation of the State Programme for Poverty Reduction of the Republic of Kazakhstan 2003-2005, first draft, (+ TOR, evaluation of regional programs and background studies) UNDP Kazakhstan.(2006) Outcome Evaluation of the UNDP Poverty Reduction Programme 2000-2005, first draft, July 2006 (+ TOR and background studies)

16