1.3 Evaluation limitations

progress reports, monitoring self-assessments such as the UNDP Results-oriented Annual Report and evaluations conducted by the country office (16 project evaluations were completed).6 The list of documents consulted can be found in Annex 4.

Key informant interviews were conducted remotely, due to the COVID-19 pandemic. In total, 105 people were consulted, including government implementing partners at national and local levels, UNDP staff, private sector and civil society partners, and staff of other United Nations (UN) agencies. The aim of the interviews was to understand stakeholders’ views on the role of UNDP in the national development agenda and its contribution to national development results through the country programme, as well as factors affecting its performance (evaluation questions 2 and 3). The list of people consulted can been found in Annex 3.

The effectiveness of the UNDP country programme was analysed through an assessment of progress made towards the achievement of expected outputs, and the extent to which these outputs contributed to the intended outcomes. To better understand UNDP’s performance and the sustainability of the results it contributes to, the ICPE examined the specific factors that have influenced – both positively and negatively – the country programme. UNDP’s capacity to adapt to the changing context and respond to national development needs and priorities was considered. In line with UNDP’s gender mainstreaming strategy, the evaluation examined the level of gender mainstreaming across the country programme and operations.

Given the scope and nature of the UNDP programme in Brazil, an electronic survey was sent to all interviewees, excluding UNDP and UN staff, to assess the quality of UNDP support and to triangulate feedback from the interviews. The questionnaire included 17 questions, covering the responsiveness and efficiency of UNDP support, and recommendations for improvement. Forty-three of a targeted 81 stakeholders (53 percent) completed the survey. The respondents were largely government staff (27 people); eight were from the private sector; two were from state-owned enterprises; three were from civil society; and three were other partners who were unspecified in the survey. Partners from research institutes and think tanks did not respond to the survey request.

The evaluation purposefully selected for in-depth review 45 of 135 projects supported by the country programme at the start of the evaluation.7 The selection took into consideration criteria such as thematic and sub-thematic focus, size (large/small expenditures), status (open/closed), geographic coverage and implementation modality. The evaluation team consulted the country office in the selection process.

The evaluation process was guided by the United Nations Evaluation Group Norms and Standards for Evaluation, as well the Ethical Guidelines for Evaluation.8 Interviewees were informed that the discussion was being recorded and interview data access was limited to the evaluation team.

The evaluation was conducted through remote interviews due to the COVID-19 pandemic. It is recognized that remote interviews affect rapport-building with interviewees, as well as willingness to participate in the evaluation. The consultation phase was prolonged due to the limited availability of key stakeholders. Several key informants were not reached despite repeated requests for interviews, and related interventions were assessed based on desk reviews of available programme documents.

6 The country programme evaluation plan for 2017-2021 included 22 project evaluations, one thematic evaluation of GEF projects and one United Nations Development Assistance Framework evaluation. By end 2020, 16 project evaluations and the GEF projects thematic evaluation were completed; 11 evaluations were quality assured by the IEO and four were rated as modestly satisfactory, four modestly unsatisfactory and three unsatisfactory. 7 This number changed during the evaluation, as some projects closed or new ones were approved. 8 See website of the United Nations Evaluation Group, http://www.unevaluation.org/document/library?categoryId=40