Transitions in the evaluation approach

2 Part 2: Endline Surveys and Special Studies

Transitions in the evaluation approach

This section summarizes the overall evaluation design and the modifications made along the way to capture Kuunika’s pivots and adaptations. Limitations to the evaluation approach are also explained. Mott MacDonald was appointed in 2016 as the independent evaluator of the Kuunika Project, with the aim of generating: (i) insights for better implementation of the project and (ii) more general lessons about how best to introduce new information technology (IT) into existing government systems in Malawi.

The evaluation originally aimed to answer five top-level questions relating to the project’s role in ‘strengthening HIV- related health data systems’. The five evaluation questions were: i) has the quality of HIV data improved? ii) has the use of that data by decision makers and practitioners increased? iii) has decision-making improved? iv) have key HIV service areas improved as a result? and v) what explains the changes (or lack of them)?

In response to the pivots and changes in the project, and intentions for a follow-on phase, the evaluators also aimed to address to supplementary questions in the synthesis stage. These were:

● How effective has Kuunika’s sustainability and COVID-19 pivot phase been? ● What should Kuunika II look like?

Our impact evaluation used a mix of quantitative and qualitative surveys conducted at baseline, midline and endline, plus a set of special studies to explore agreed areas of the theory of change more deeply.

The baseline evaluation conducted in August 2017and the midline evaluation was conducted in April 2019. This section of the report (Part 2) sets out the findings of endline evaluation – which included a repeat of baseline/midline mixed method data collection exercise and three ‘deep-dive’ special studies. The findings from the endline evaluation, together with reflections from the earlier evaluations, have formed the evidence base for the Part 1 synthesis and recommendations.

To capture the changes in the project, and respond to recent restrictions imposed by the COVID-19 pandemic, the midline and endline surveys involved some modifications from the baseline. Although, technically, this was a deviation from evaluation and research best practice, this responsive approach allowed us to maximize the lesson-learning value of the evaluation.24 The main modifications to the evaluation approach at midline and endline were:

● The midline covered a different set of districts to the baseline. In 2017, we covered Blantyre and

Machinga Districts as ‘treatment’ districts and Balaka District as a comparator. At the request of the

BMGF and Kuunika, we swapped Machinga district for Zomba District at midline.25 Zomba District had become a district of focus for an accelerated core package of interventions in November 2018, so it was agreed that evidence from Zomba District would be more useful for informing decisions on Phase 2 of the project. ● The endline evaluation has covered all four districts,26 including both Machinga and Zomba. This has allowed us to retain as much time series data as possible. ● The international team did not travel to Malawi for the final round of endline data collection. Due to

COVID-19 travel restrictions and safety measures, the endline data collection had to be overseen and supervised remotely. In practice, this meant: the KAP survey was conducted face-to-face by local enumerators (with appropriate safety measures in place); focus group discussions were

24 This responsiveness might also be seen as consistent with BMGF’s preference for evaluation work to support ‘co-creation’. 25 The districts were originally selected to be as similar as possible in influential contextual factors such as district-level administrative capacity, urbanisation and geographical accessibility of facilities and HIV/AIDS prevalence. The new midline district, Zomba, was similarly consistent. 26 As a result, the 2021 KAP survey was larger than the previous two rounds, with N = 220+ in 2021 compared to around 180 in 2017 and 2019.