2.4 Challenges and limitations

• Most significant change analysis of participatory narrative survey data to understand to what extent perceived psychological and behavioural change was facilitated by UNDP interventions, and to identify factors affecting the UNDP performance; dedicated

SenseMaker© software was used to produce quantitative data, uncover relationships and patterns embedded in youth stories and identify typical and atypical stories, which were interpreted jointly with project stakeholders • Theory-based analysis of actual versus intended results and influencing factors • Analysis of performance against the OECD/DAC criteria

To analyse the level of gender-related approaches and results, the IEO Gender Results Effectiveness Scale was employed (see figure 4). Gender marker data were used for analysis of gender programme expenditures against commitments made, and sex-disaggregated data were assessed where available.

Informed by a systems approach, evidence was contrasted and patterns synthesized into key findings to report on the key evaluation questions, framed around a triangulation matrix (see annex 12). Higher-level conclusions and forward-looking recommendations were derived from this analysis.

FIGURE 4. IEO Gender Results Effectiveness Scale

Gender Negative Gender Blind Gender Targeted

Gender Responsive

Gender Transformative

Result had a negative outcome that aggravated or reinforced gender inequalities and limiting norms

Result gave no attention to gender, and failed to acknowledge the different needs of men, women, girls and boys, and other marginalized populations Result focused on the number of women, men, or marginalized populations that were targeted (e.g. 50/50 representation)

Result addressed the differential needs of men, women, or marginalized populations and focused on the equitable distribution of benefits, resources, status, rights, etc. but did not address root causes of inequalities Result contributed to changes in norms, cultural values, power structures and the roots of gender inequalities and discriminations

Source: Adapted from the Evaluation of UNDP Contribution to Gender Equality and Women’s Empowerment, IEO, UNDP, 2015

COVID-19 and remote work. This evaluation was conducted under the challenging circumstances created by the COVID-19 pandemic. This meant that the evaluation team was unable to travel and therefore collected and analysed data remotely. Another implication of the pandemic was that SenseMaking, usually carried out as an in-person workshop, had to take place remotely via Zoom or other means, which excluded some potential participants due to low literacy skills and lack of connectivity.

While these extraordinary circumstances presented some limitations, the evaluation was still able to respect evaluation norms and professional standards.

Data scarcity. The main challenge to the exercise was the paucity of monitoring data (poor age/sex disaggregation, little information on quality of services, etc.), along with the unavailability of some key project documents, periodic reports and in some case the absence of beneficiary lists. The availability of documentation varied across different interventions, making it difficult to identify the results UNDP attributes to its youth economic empowerment projects. To mitigate these challenges, and for triangulation purposes, the evaluation team broadened the scope of its secondary data review by including external assessments and evaluations to cross reference internally available data and validate findings. It also used additional data collection tools such as the SenseMaker and a youth organization survey as a mitigation strategy for data paucity.

Low evaluability. The youth economic empowerment evaluability was moderate to low, namely because monitoring data on results at intermediate steps of the results chain was lacking, as were impact-level data. More importantly, a clear theory of change for youth economic empowerment and a corporate results framework to frame the evaluation were absent. The evaluation worked with the best available data but recognizes quality and coverage issues. To try to address this limitation, the evaluation team reconstructed a theory of change and recruited five seasoned evaluators who are specialists in youth economic empowerment. The team also relied on the support and guidance of an advisory panel composed of recognized specialist practitioners and academics on youth economic empowerment.62

Mapping of youth economic empowerment interventions. The evaluation faced challenges in identifying and mapping UNDP work on youth economic empowerment. The use of the UNDP Transparency Portal and youth marker in the UNDP Atlas software (c.f. finding on monitoring and evaluation) did not yield a full list of projects promoting youth economic empowerment which could have been used as a sampling frame. Countries and projects sampled are indicative of the extent of youth economic empowerment work across UNDP programming in 135 countries; the evaluation makes no claim to the sample being fully representative. To assess programme performance on the ground, the evaluation drew a purposive criterion-based sample of UNDP-supported initiatives taking into consideration regional distribution, type of youth economic empowerment intervention, signature solution, gender marker and country characteristics, among others. This approach allowed for the identification and selection of information-rich cases related to the phenomenon of interest while being cost- and time-effective. Purposive sampling allowed the evaluation to identify key patterns and emerging trends and draw valid conclusions, while not claiming representativeness in statistical terms (non-probability sample).

Compressed time frame. Time constraints limited the extent of data collection and analysis. Notably, the SenseMaker exercises in Somalia and Tajikistan had to be dropped due to limited time to arrange interpretation services. Data collection was conducted over a short period of six weeks in June/July 2021 which coincided with summer holidays for some key informants who were therefore not available for interviewing.

62 The advisory panel provided support for the review of the terms of reference and the evaluation theory of change, the development of the signification framework for the SenseMaker as well as for the review of the final evaluation report.