Table 1: Summary of key findings from the special studies

– Findings from KIIs and FGDs suggested however that HMIS/ DHIS2 data viewed is still perceived to be of limited value to target users – especially at district and facility levels.

● Have attitudes and practices around decision-making improved? – With regards sub-national user engagement, there was little sign that perceptions of ‘vertical data extraction’ have changed since baseline. Despite multiple examples of Kuunika engagement with districts, interviews suggested most respondents at this level felt they were primarily seen as sources of data, rather than as collaborators in designing digital systems for use in service delivery. – With regards data quality and use at sub-national levels, there was some evidence from the KAP survey of improved perceptions of data quality. However, there was little evidence of sustained improvements in data quality control practices – especially with the onset of COVID-19. Evidence of sustained improvements in data use practices was mixed: the KAP survey pointed to regular data use for reviewing patient records and stock balances; however, our findings also indicated an overall decline in the frequency of data use activities since baseline.

● Have key [HIV] service areas improved as a result? – In light of the project pivots and adaptations, the evidence base for Kuunika’s contributions to intended HIV service delivery outcomes remained fragmented and inconclusive.

Key findings of the special studies

Key findings from the three special studies on the pivot to COVID-19, digital health governance, and the role of Kuunika at the district level responses are summarized in Table 1 below. The findings are organized by key achievements, key challenges and key lessons.

Table 1: Summary of key findings from the special studies

SPECIAL STUDY KEY ACHIEVEMENTS KEY CHALLENGES KEY LESSONS

1. COVID-19 monitoring and the digital surge

2. Governance the Regulatory &

Policy

Essentials

• Suite of 12 COVID-19 products developed rapidly & collaboratively • Products led to a reputational boost for Kuunika, and demonstrated the convening power of the DHD • Kuunika successfully supported functions of the public health emergency operations centre &

Pandemic Task Force • With respect to the COVID19 products, some weakness were found re: district engagement & end user consultations; reporting burdens; system errors & updates; accessible technical support; hardware & connectively challenges • Study identified opportunities for better engagement of e-

Government technicians, capacity building and wider partner engagement –especially at sub-national levels

• Significant contributions to the

Expanded Health Data

Exchange, and a national digital health vision based on the new

Digital Health Strategy and modular OpenHIE framework • Governance positions on policy and standards compliance in Digital Health

Division have yet to be filled • Conceptualization of the digital health ecosystem needs to extend beyond technology • Key legislation & regulations are in place, but ongoing attention must be given to data protection & cybersecurity threats

3. Kuunika and the

Districts

• Kuunika has invested significant time, effort & resources in district-level capacity development - especially before 2019 • The Blantyre Prevention

Strategy is a potential best practice model – it builds on lessons from Kuunika and is fully embedded in district structures • Districts remain ‘the missing middle’ - evidence suggests districts are still neglected in digital health development • Successive evaluations point to continued issues of system fragmentation, and weak / unsustainable data quality and data use initiatives • Kuunika’s focus on the vertical HIV program as the

‘use case’ meant there was limited opportunity for districts to engage • Customised / differentiated data use approaches are needed at district and national level

Introduction

The Kuunika: Data for Action Project was launched in November 2016 with the aim of improving the planning, performance and quality of health services in Malawi through better digital information. The Kuunika Project is a US$10 million, four-year project funded by the Bill & Melinda Gates Foundation.

In 2016 the Bill & Melinda Gates Foundation (BMGF) appointed Mott MacDonald as the independent evaluation partner for the Kuunika Project, with the aim of generating: (i) insights for better implementation of the project and (ii) more general lessons about how best to introduce new information technology (IT) into existing government systems.

A primary objective the Kuunika Project was to demonstrate the effectiveness of a customized package of interventions in the context of the HIV /AIDS programme ‘use case’. In line with this objective, the evaluation design aimed to address five top-level questions relating to the project’s role in ‘strengthening HIV- related health data systems’.4 The five evaluation questions were: i) has the quality of HIV data improved? ii) has the use of that data by decision makers and practitioners increased? iii) has decisionmaking improved? iv) have key HIV service areas improved as a result? and v) what explains the changes (or lack of them)?

This Final Synthesis Report is the last report in a series of reports covering Mott MacDonald’s baseline, midline and endline evaluations. The Final Synthesis Report is accompanied by a set of separate ‘deep dive’ Special Study Reports that investigate Kuunika’s recent pivot to respond to COVID-19, digital health governance themes and the role of the project at district level.

PART 1 of this report focuses on a synthesis of key findings from the endline evaluation enquiry, including the special studies. Section 1.1 examines the original project design and theory of change developed by the evaluators at baseline to inform the overarching evaluation design. Section 1.2 describes how the project evolved over time by navigating three key ‘adaptation points’. In Sections 1.3 and 1.4, the evaluators draw on the evidence synthesis to identify lessons emerging from the independent review of the ‘Kuunika journey’, along with a set of residual challenges. These lessons and challenges have informed the evaluators’ reassessment of the hypotheses underpinning the original theory of change (see Annex 1), and lead to the presentation of a revised theory of change in Section 1.5. Part 1 of this report concludes with a set of eight recommendations for the next iteration of the Kuunika Project, as well as a list of considerations for BMGF (and other donors) when designing similar programs of this nature.

PART 2 of this report provides a more detailed account of the evidence base that contributed to the Part 1 synthesis and recommendations. In this section, we present the findings of the endline evaluation – this repeated the mixed method data collection exercise conducted at baseline and midline. Sections 2.1 and 2.3 describe the responsive evaluation framework and methodology for the endline evaluation exercise. Section 2.3 presents the triangulated quantitative and qualitative findings from the endline data collection round. Part 2 of this report concludes with a summary of the main findings from the three special studies.

The Annexes to this report provide more detailed accounts of the evaluation design, the evidence and literature base for the theory of change, the evaluation data collection tools and instruments and more detailed quantitative and qualitative findings. Annex 6 contains an important list of lessons for designing future evaluations of this nature.

4 The titles of Mott MacDonald’s baseline and midline evaluation report reflected this focus.