Figure 2: Overview of key Kuunika achievements at endline

Figure 2: Overview of key Kuunika achievements at endline

Evidence-based lessons and recommendations

The endline evidence synthesis has allowed the evaluators to identify a number of lessons arising from the ‘Kuunika journey’. These largely relate to the three Kuunika pillars of: improving data supply; increasing data demand and use; and strengthening data governance – although there are some important generic lessons and lessons for evaluation methodologies (see Annex 6). Key lessons for further reflection include: ● A need to maintain a strong focus on the supply of quality data at each system level as a foundation for consistent data use for decision-making – there remains evidence of weak data quality in key data repositories ● A need for more differentiated data use strategies at each system level to support a more systematic progression towards an inclusive data use culture ● A need to advance consensus-building on a joint Roadmap for operationalising and resourcing the National Digital Health Strategy – this might also require more rigorous thinking on the challenges of maintaining data quality across mixed / hybrid reporting systems ● The benefits of ‘widening the lens’ on digital health governance to embrace inter-ministerial collaboration, investment in digital health governance capacity and a more comprehensive view of the digital health ecosystem. The final evaluation has also spotlighted some residual challenges in progress towards the project goal of improving the planning, performance and quality of health services through better digital information. While some of these challenges may now be beyond the scope of the Kuunika Project, they are included in this report to give ‘voice’ to the full range of stakeholder perspectives, and to inform future program design. The evidence-based challenges identified at endline largely related to persistent ‘weak links’ in maintaining data quality and data use at district level and ‘points of care’, and concerns expressed by district-level cadres that they were not regarded as “equal digital health partners”. Some weaknesses were found in digital health governance - the ongoing challenge of maintaining robust data protection measures and protection against cybersecurity threats in this ‘immature’ system setting was also noted.

The evaluators have drawn on a revised theory of change, lessons learnt and the overall evaluation findings to present eight recommendations for the next iteration of the Kuunika Project. These are detailed in Section 1.6 of this report and are summarized in the Box below.

RECOMMENDATIONS FOR A FUTURE KUUNIKA PROJECT

1. Continue to back the National Digital Health Strategy (2020-2025), and the modular approach to building the digital health architecture 2. Prioritize effective mechanisms for convening and coordinating partners around a shared digital health vision 3. Extend the Digital Health Division’s reach and links - up, down AND across 4. Engage districts more in digital health planning, convening and design processes 5. Invest in digital health governance capacity 6. Broaden the conceptualization of the ‘digital health ecosystem’ to engage with the ‘real-world’ political economy and institutional factors that can ultimately determine digital uptake and data use for decisionmaking 7. Have a sustainability and exit plan from the outset to engage with the human and financial resourcing challenges of sustainable digital health solutions 8. As the digital health system progresses towards Shared Electronic Health Records (EHRs), give particular attention to issues of data protection, privacy, and cybersecurity threats

The evaluators have also identified a number of considerations for BMGF and other donors to keep in mind when designing other programs of this nature. Key considerations refer to: a) the need to engage with system complexity and the widespread fragmentation of digital health initiatives in low-income settings; b) the need to strive for more aligned, multi-component and multisectoral approaches to digital health development, while remaining flexible and adaptive; and d) the need to work with other development partners to apply the principles of aid effectiveness and sustainability from the outset.

Part 2: The Evidence Base - Key Findings from Endline Assessments

Key findings by evaluation question

Endline data collection was completed in September-October 2021 to inform the time series analysis in line with the original evaluation design. Data collection at endline covered: i) 20 health facilities across four target districts (Balaka, Blantyre, Machinga, Zomba); ii) District Health Management Offices (DHMO) in the target districts; and iii) the Ministry of Health (MoH), Lilongwe. Endline evaluation findings were validated by triangulating findings from across data sources. As at baseline and midline, the endline analysis focused on addressing the core set of theory-based evaluation questions. Key findings from the triangulated analysis by evaluation question are summarized below.3

● Has data quality improved? – The endline DQA found that, of the three proxy indicators selected at baseline, only one (# pregnant women testing HIV positive) was reliably available on HMIS / DHIS2. Data quality for this indicator had improved in each sampled district since baseline. – Loading times: HMIS / DHIS2 web-page loading times had remained stable since early improvements in 2017-18. – The user assessment found strong examples of clear and user-friendly data visualisations and dashboards on the DHIS2 and OHSP platforms, although there were also some minor issues with data consistency and labelling. ● Has data use increased?

– KAP survey and DQA findings suggested monthly use of HMIS / DHIS2 data had increased over time - across our sample, there were around twice the number of unique monthly users in

September 2021 as in September 2018. However, other data use indicators showed relatively little change over this time period.

3 See Annexes 3-5 for key findings by data collection method.