Table 4: Summary of limitations of the endline evaluation design and methodology

conducted ‘semi-remotely’, facilitated by a local team member; and key informant interviews were largely conducted virtually from the UK. The final data quality audit was conducted as a desk review in UK, and focused primarily on the DHIS2 user interface; at endline it did not include a systems assessment. With regards the special studies, these were originally intended to explore key areas of the Phase 1 theory of change. However, the final selection of study questions aimed primarily to understand the implications of key issues and events during project implementation, with the ultimate aim of informing a revised theory of change.

The limitations to the evaluation design and methodology at endline are acknowledged. They are systematically described in the table below, along with mitigating action taken.

Table 4: Summary of limitations of the endline evaluation design and methodology Limitation Description Implication and Mitigating Action

Small sample sizes Project design and budget limited feasibility and value of larger sample size

Some changes in panel sample of districts over time

Some KAP survey questions omitted at endline Original intended phased roll-out of implementation across identified districts was dropped as project approach evolved over time Possible programming errors and limited remote real time QA of fieldwork resulted in some data collection errors Multiple respondent categories imply small numbers so that quantitative data is not statistically robust. We use ‘small n’ evaluation methodology and aimed to ensure the validity of findings through triangulation techniques.27

Original intention to treat Balaka as a comparator district could not be carried through. Some implications for the time series analysis. To maintain rigour, final analyses did not include disaggregation of data by district. Implications for time series analysis. Only findings that endline survey findings that are fully evidence based have been included in this report.

Availability of key stakeholders to participate in remote KIIs

On-site DQA data verification against source documents and system assessment not possible

COVID-19 travel restrictions Despite extensive sample frame of contacts, there were significant problems in setting up KII by phone, particularly with facility and district staff

COVID-19 safety measures required significant modification of original facility-based DQA method

Mott MacDonald’s duty of care responsibilities and COVID-19 response meant safety was prioritized and international travel was strictly limited during 2020 and 2021 Delays in completing target numbers and range of KII participants. FGDs at district level facilitated by national team member filled some gaps. Limited interview data for endline and special studies (especially Special Studies 2 and 3) were mitigated through intensification of document and secondary data reviews. An alternative approach of photo-capture of register pages and summary forms by the enumeration team during KAP survey was introduced; however, some blurred photos limited number of observations. The intended omission of system assessment component reduced qualitative data, but this was partly compensated for by ad hoc remote KIIs. International team members did not visit sites or interview stakeholders in person. However, trainings, regular debriefs and key informant interviews were conducted remotely. With minor exceptions, the same evaluation and enumerator teams that were used for baseline and midline evaluations were used – they were, therefore, highly familiar with the evaluation context and methodology, and experienced in using the evaluation data collection tool-kits.

27 Addressing attribution of cause and effect in small n impact evaluations: towards an integrated framework. H White & D. Phillips, 2012