Independent Evaluation of Kuunika: Strengthening HIV Related Health Data Systems

Acknowledgments

This special study was undertaken as part of the evaluation carried out by Mott MacDonald Ltd and the Centre for Development Management (CDM) on behalf of the Bill and Melinda Gates Foundation.

This Special Study 3 was written by Dr Janet Gruber, with inputs by Rachel Phillipson and Professor Maureen Chirwa.

We are extremely grateful to all the key informants who gave time from their busy schedules to talk to us about Kuunika, decentralization and District health services.

Special thanks to the Centre for Development Management (Malawi survey company): Bright, Wingston, (enumerators).

We are also grateful for support and comments from Cooper Smith

Issue and Revision Record

Document reference:

Information class: Standard

This document is issued for the party which commissioned it and for specific purposes connected with the above-captioned project only. It should not be relied upon by any other party or used for any other purpose.

We accept no responsibility for the consequences of this document being relied upon by any other party, or being used for any other purpose, or containing any error or omission which is due to an error or omission in data supplied to us by other parties.

This document contains confidential information and proprietary intellectual property. It should not be shown to other parties without consent from us and from the party which commissioned it.

Abbreviations and Acronyms

BMGF Bill and Melinda Gates Foundation

BNA Bottleneck Analysis

CBO Community-based Organization

CHAI Clinton Health Access Initiative

CMED Central Monitoring and Evaluation Department (MoH)

DCSA Disease Control Surveillance Assistant (formerly HSA)

DDP District Development Plan

DHIS2 District Health Information System 2

DHO District Health Office or Officer

DHMT District Health Management Team

DIP District (health) Implementation Plan

DNHA Department of Nutrition, HIV and AIDS (previously the Department of HIV & AIDS)

DNHA-MIS Department of Nutrition, HIV and AIDS’ Management Information System

DQA Data Quality Audit

EMR Electronic Medical Records/Register

FGD Focus Group Discussion

HMIS Health Management Information System

HSA Health Surveillance Assistant

ICT Information and Communications Technology

IDSR Integrated Disease and Surveillance Response

KAP Knowledge, Attitudes and Practice

KII Key Informant Interview

LAHARS Local Authority HIV and AIDS Reporting System

M&E Monitoring and Evaluation

MEHIS Monitoring & Evaluation & Health Information Strategy (2017 - 2022)

MEL Monitoring, Evaluation and Learning

MoH Ministry of Health

NAC National AIDS Commission

NDHS National Digital Health Strategy (2020 - 2025)

NGO Non-governmental Organization

NHISP National Health Information System Policy (2015)

OHSP One Health Surveillance Platform

SDG Sustainable Development Goal

SEP Socio-economic Profile

SS Special Study

SWAp Sector-wide Approach

UHC Universal Health Coverage

Executive Summary

This Special Study forms part of the independent endline evaluation of Kuunika. It was requested by the Bill and Melinda Gates Foundation.

Certain factors relevant to decentralization of the health system and digital data architecture and use are beyond the control of any project, including Kuunika Such factors include the longstanding human resources for health challenges, which are especially acute in rural areas, the fact that many Data Clerks, on whose shoulders rest much of the daily task of data entry, receive too little training and other support and are often co-opted from other roles. Moreover, decisions at national government level as to the extent and detail of any devolution of financial autonomy to Districts and sectors are beyond any project's control.

Therefore, in this report we examine the space available to Kuunika in terms of its engagement with the District, Health Facility and community levels, how it used such opportunities to support those health service planning and delivery structures specific to application and use of the DHIS2 digital data system. In addition we consider how Kuunika has been able to support digital data use at national level. We examine if the project has effectively engaged with Districts and lower levels to optimize impact of its activities and support. We consider such work throughout the Kuunika pivots and its Covid-19 support, thereby reviewing as from the 2016 grant proposal, as well as reinterrogating 2017 baseline and 2019 midline findings specific to District inputs and broader issues of decentralization.

We also consider the principles of aid effectiveness and sustainability: if the commitment to decentralized health structures and effective ownership and/or use of digital data at that level is limited from the side of national government, what traction might any project have? How might a project then best engage more with existing District systems in a predominantly horizontal approach, so as to engage more directly with health planning and service delivery challenges and the effective use of digital data to address those?

One finding is that attention by the project partners to the specifics of decentralization and the ramifications for Kuunika has throughout been limited. As a result, the independent evaluation of Kuunika has hitherto not examined decentralization issues in much detail.

Key findings

1. Decentralization of the health system and digital data is beyond the control of any project, including Kuunika Information is that there are currently (mid-2021 onwards) changes being made at national government level specific to what might be seen as a (re) centralization of digital health data management and (re) focus on DHIS2. Thus the Digital Health Division has been moved back into the MoH Department of Planning and Policy Department.

2. The move might limit, even reduce, districts' and health workers' capacity to review and make speedy use of data; such moves might also have impacts on the building of digital data systems and knowledge culture at District and facility levels.

3. The partial, piecemeal and stop-start implementation of the government policy of decentralization is viewed as a cause of poor governance in the health sector Governance challenges are a significant barrier to achieving a more effective and equitable health system in three key domains: accountability (enforceability; answerability; stakeholder-led initiatives); health resource management (healthcare financing; drug supply); influence in decision-making (unequal power; stakeholder engagement).

4. Districts are the 'missing middle' in many respects in terms of GoM and donor partner engagement with the health sector; there is increased focus on community engagement (and quite rightly, not least for equity considerations), yet Districts are the entry point to the great bulk of health services delivered to Malawians.

5. When Kuunika was being designed, the GoM said it was decentralizing the health system, but most decisions and human resource management continued to be made at the central level. There is a centralized 'push system' in health. Most projects at District level have had to work very closely with the MoH, even if more focused on the Districts; this has been true for Kuunika

6. The MoH is the owner of Kuunika - this has been made clear from project inception. Therefore, has the project made best use of the space available within the predominantly centralized health systems and structure to engage with Districts to build sustainable, standalone use of digital data? The answer is, on balance, no.

7. The cumulative experience of the Kuunika evaluation shows that there needs to be greater genuine ownership of data at the District level and effective use of DHIS2 as a platform for more effective evidence-based planning. This necessitates buy-in from all partners working in any one District and nationally, challenging in a donor landscape currently as fragmented as Malawi; this has obvious implications for effective coordination of digital health initiatives. The District Health Offices should ideally be integral partners in data collection and use

8. Kuunika has invested a very great deal of time, effort and resources into District level capacity development, training on and access to DHIS2 and digital data hardware and systems (e.g. dashboards, the mobile App, Cluster meetings). This was most apparent before 2019, at which point changes in consortium partners and project management, coupled with the sustainability pivot, led to Kuunika being seen as becoming more distant from the Districts (except in Zomba). More recent support to horizontal engagement, e.g. through helping to facilitate setting up Cluster meetings, are often not recognized as having Kuunika inputs, but are seen by those involved as useful channels for debate and decision-making processes.

9. The Covid-19 pivot, centrally facilitated by the MoH with Kuunika and other partner support, has enabled considerable progress to be made by Malawi in terms of tracking the pandemic and developing a response. At District level there are mixed views as to how much such data collection has supported service delivery.

10. Despite being defined in terms of being a District-focus programme in the early days, Kuunika has given relatively limited attention to planning effectively for how it might most coherently and comprehensively support decentralized structures and systems and provide optimal inputs at District level. for access to and use of digital data, through the DHIS2 platform. The temporary involvement of the Districts in Kuunika planning after the first pivot appears to have been short-term and never properly integrated in project planning and processes.

11. Kuunika might be most usefully regarded as sitting in the ‘functions and capabilities’ space, its contribution to decentralization of health services lying in its ability to empower Districts via access to better data to plan, manage and deliver services.

12. The decision to focus on HIV as the data use case, a vertical system owned at the central level (and where key datasets collected and managed by the DNHA were not uploaded to DHIS2), meant there was little space for the District level to engage as an equal partner in management and use of digital data. Initially the Kuunika consortium was almost a parallel MoH, without any attention to decentralized health and data. Kuunika moved also; at one time it was in the HIV Department.

13. Kuunika was somewhat designed in a vacuum, without thinking of communication channels between the project and Districts. The initial big focus in Kuunika was data systemsnot which entity/individuals had access, ownership, etc. Later focus was more on patient outcomes and use of data to support service delivery. That necessitates proper District buy-in if improvements in service delivery are to be achieved - and that step has not been properly taken by the project.

14. At the end of Kuunika phase 1, there is little concrete evidence of sustained, systemic improvements in data use knowledge culture at District level, or of key planning and service delivery documents such as DIP being progressively informed by quality data derived from DHIS2. There continue to be considerable problems of access to DHIS2, of capacity to navigate its programs, of identifying, analysing and using quality data.

15. The existence of DHIS2 data 'super users' demonstrate the possibilities for building a knowledge culture based on interest twinned with capacity. The question is how to maximize such potential without thereby placing unrealistic burdens on individuals.

16. The Blantyre Prevention Strategy is said to be designed based on lessons learned regarding Kuunika District gaps. This new program has the DHO as the lead, from the start. 'We re-thought the process, based on Kuunika challenges with decentralization.'

Overview of three key issues

Decentralization, Districts & Kuunika

• Neither a decentralized 'decision space' for health planning nor functions + capabilities have been effectively supported

• Development of digital health guidelines + systems should be supported more consistently at the District level

• That support should be based on a more coherent approach to the development of an organizational knowledge culture: starting with the users, not the system

• The Covid-19 pivot has not resulted in clear wins for DHOs + Health Facilities (+ perhaps communities): primarily seen as data extraction without effective support.

Sustainability

• Kuunika's key partner is the MoH; however, sustainability of digital health systems requires support to Districts and lower levels, as the central point of service deliveryfor HIV as well as universal health coverage

• The project support to the development of the MoH Digital Health Division represents not only a solid + (it is hoped) sustainable gain; the DHD could also serve as a central point for more aligned + harmonized MoH + partner engagement in work on digital health data systems + use, e.g. if the 'SWAp revival' concept note results in more coordinated inputs

• Kuunika inputs to e.g. the 2020-2025 National Digital Health Strategy represent normative + longer term contributions that can underpin more sustainable digital data architecture

Aid effectiveness

• There continues to be data extraction, imposition of donor partner data systems + indicators, lack of DHO + Council engagement in planning + creation of context-specific plans.

• There is scope (+ urgency) for greater coherence + genuine partnership at District level in digital data collection + use.

• There is fragmentation of effort, due to lack of donor co-ordination across the many health programs and projects. At District level, DHO, Council and Health Facility staff are overwhelmed by multiple demands + reporting formats. This affects the efficacy and efficiency of DHIS2. Alignment of all partners, with the active engagement of District actors, is essential.

• Districts continue to be the 'missing middle' in terms of the digital data architecture

Recommendations for Kuunika phase 2

Recommendation 1: engage throughout with Districts - leadership, ownership and governance

Rationale: the ultimate goal of any data system should be to deliver optimal health services that lead to improved patient outcomes. That necessitates proper District buy-in, which was not embedded into Kuunika from the outset.

Recommendation 2: work for greater aid effectiveness - alignment with other partners working on digital health data

Rationale: information from national level respondents is that a 'Sector-wide Approach (SWAp) Revival' Concept Note is in development. There is increasing emphasis on maturing away from a myriad of pilots and toward proven and scaled solutions built on common standards within an architecture.

The consensus appears to be that any such action would seek to re-introduce SWAp principles to alignment, joint planning, working and Monitoring, Evaluation and Learning (MEL), but not (at least initially) any financial disbursements to government entities. Investments and implementation for digital health data require far more harmonization. Application of a number of SWAp principles could potentially link disbursements to District performance, monitored using digital data. This would, however, necessitate genuinely effective support and training to all those engaged with collecting and using data to plan and deliver services.

Kuunika (and the Bill and Melinda Gates Foundation) should continue work with the MoH to support greater alignment and harmonization of all partners' digital health data interventions.

If there is a SWAp-light framework developed for digital health data, Kuunika should consider a role as an integral partner, calling upon cumulative project experience, expertise and its position at the center of national developments.

Recommendation 3: continue to support the MoH Digital Health Division

Rationale: the Digital Data Division should continue to receive Kuunika support; it is the site of government technical capacity. In its inception phase Kuunika 2 should plan for greater DDH coordination with and support to District digital health data systems.

Recommendation 4: ensure digital data systems are designed with users in mind and work to maximise opportunities for all levels to have access to participation, training and ongoing support

Rationale: systems should not be designed without properly thinking of who will use them and how, and the capacity development each individual cadre will need. With hindsight, the adoption of HIV as a data use case for Kuunika may have limited scope and flexibility - collection of HIV data was and remains tightly managed at the national level. Districts and Health Facilities did not have oversight of such data, or effective ownership.

Recommendation 5: build in sustainability from the outset

Rationale: while there is increasing resistance to a 'proliferation of pilots' in the digital health data sphere, Kuunika 2 has options to build on the foundations not only of the project, but to engage closely with the Blantyre Prevention Strategy, which is focused on building Districts systems, capacity and ownership. Such relationships, allied to any development of greater partner alignment and harmonization, could enable not only economies of scale but evidence-based prioritization of interventions proven to be effective.

Recommendation 6: have more focus on equity aspects of digital data systems

Rationale: data are never neutral. Just one point is disaggregation of data - their collection, their identification, their analysis and their use. WHO, UN Women, many civil society organizations and

others continue to press for greater equity of data disaggregation, and greater application of such principles in the context of digital data.

1. Introduction

In October 2016, the Bill and Melinda Gates Foundation (BMGF) approved Kuunika: Data for Action, a $10 million, four-year program of support to the Government of Malawi’s Ministry of Health (MoH), to improve the planning and performance of HIV services in Malawi through the use of digital health data. HIV was chosen as a data use case for the project.

Kuunika's Theory of Change can be summed up as:

IT infrastructure investments lead to improved ‘data outputs’: evident and measurable improvements in the amount of time the databases are available to the user (availability) and in the ease with which the data can be retrieved, combined and viewed (accessibility). Many new ways of combining and analyzing data thereby become possible. This potential is realized through training and capacity building to provide users with skills and incentives to use the new data systems. The creation of MoH-wide data governance structures harmonize and maintain data standards, underpinning users’ trust in the quality of the data and further encouraging its use.

Mott MacDonald was appointed at the same time as the independent evaluator of Kuunika, with the aim of generating lessons from the program about how best to introduce new information technology (IT) into existing government systems. There have been three iterations of the evaluation: the 2017 baseline, the 2019 midline and the 2021 endline, as well as a Program Implementation Review in 2019. The three deeper dive Special Studies were all agreed with BMGF and Kuunika. 1

The evaluation has throughout aimed to answer five top-level questions:

As a result of the Kuunika project…

1. Has the quality of [HIV] data improved?

2. Has the use of that data by decision makers and practitioners increased?

3. Has decision-making improved?

4. Have key [HIV] service areas improved as a result?

5. What explains the changes (or lack of them)?

The endline and the three special studies additionally address the following two questions:

6. How effective has Kuunika’s sustainability phase been?

7. What should Kuunika II look like?

1.1 Special Study 3

This study examines the extent to which the Kuunika project may have responded to, or influenced, decentralization processes and outcomes in the Districts where the project has been implemented. The study will consider whether and how the project has supported digital Health Management Information Systems (HMIS) expansion, digital data use, overall knowledge management for enhanced health service delivery at the district level and whether decentralization processes have had any bearing on such work.

The study explores Kuunika’s relationship with the Districts and considers the (theoretical and evidenced) potential for digital Health Management Information Systems to contribute to effective decentralization of the health system.

1 See the endline report for detailed discussion of the history of the independent evaluation since 2016.

Over-arching Special Study 3 topics are:

1. How has Kuunika addressed and involved the Districts in project delivery?

2. To what extent have Kuunika outputs been adopted and ‘capabilities created’?

3. Can we look at decentralization and use of data through the lens of Kuunika?

4. To what extent has the pandemic pivot played a part in decentralized data systems?

Key informant questions include: 2

• Has the key informant's organizationseen any changes in the past 5 years in terms of health system decentralization processes?

• What is the current status of decentralization in the context of health service financing, governance, data collection and use and service delivery? Challenges?

• The role of digital data: how can data be used at District and national levels to plan and improve health service delivery?

• Has decentralization specific to health (however limited in scope) had any impacts on the key informant's access and inputs to, and use of, digital health data and data systems in decision making?

• What autonomy and authority do Districts have specific to health planning and service delivery in the context of decentralization?

• (How) has Kuunika addressed Districts’ specific digital data needs and challenges?

• (How) has Kuunika supported Districts in management of digital health data?

• To what extent have Kuunika outputs been adopted and ‘capabilities created’ for digital health data use - at the District level?

• Has Kuunika been agile in identifying and responding to decentralization activities that may have an impact on project work?

• Any changes due to the MoH and Kuunika response to the pandemic, specific to digital data and the role and remit of Districts?

A number of Kuunika evaluation hypotheses are relevant for this special study:

1. The degree of organisational and political decentralization can affect use of evidence in decision making (hypothesis 5)

2. Policy making is often messy and opportunistic - ‘a disorderly set of interconnections and back and forthness' (links into hypotheses 7, 9 and 17)

3. Hierarchical management of information and/or organisational silos can limit access to data and its use. Divisions of responsibilities and ‘silos’ can also limit consideration of evidence (hypothesis 15)

4. Individuals are empowered through access to data (hypothesis 18).

The various iterations of the Kuunika Theory of Change as developed by the evaluation team are also relevant, in order to consider the extent to which not only the project but also its evaluation addressed decentralized processes.

The first two deliverables in the Theory of Change address 'improved system architecture' and 'enhanced accessibility and useability' - key for decentralization. Organizational change is also essential if health systems strengthening specific to digital data collection and use is to be achieved.

The outputs set out in the ToC: 'new data services' and 'new data use skills', as well as the planned outcomes, have been reviewed in the context of decentralized systems and structures

2 See Annex 2 for KII discussion guides.

Mott MacDonald | Independent evaluation of Kuunika: strengthening HIV related health data systems

The draft final ToC shown below was developed by the evaluation team in September 2021; any further refinements will be discussed in the final draft of this Special Study report and in the final full evaluation report

Just to note here that the iterations of the evaluation Theory of Change do not explicitly address decentralized health systems or the role of digital data within those, due to being responsive to the project parameters. However, deeper attention could have been productive; just one example - the first two deliverables - 'improved system architecture' and ‘enhanced accessibility and usability' require effective partnership at all levels of the system in order to contribute optimally towards outputs and outcomes.

Figure 1: Final ToC 092021 reflecting 2018 project pivot, 2019 sustainabilty phase + 2020 Covid18 response

1.2 Special Study 3: approach and sample

The qualitative findings presented here are based on triangulated data collection from literature review, key informant interviews (KII) and focus group discussions (FGD), linked to contribution analysis and examination of the Kuunika evaluation Theory of Change and hypotheses.

The literature review is available upon request from Mott MacDonald. See Annex 1 for references to documents reviewed. It covers the following topic areas:

1. Definitions + debates: decentralisation, devolution, deconcentration

2. Relevant Government of Malawi (GoM) documents

3. District Development Plans and District Implementation Plans (DDP and DIP)

(A number of DDPs and DIPs for the sample Districts have been reviewed for 2017 - 2022 (DDPs) and 2019 - 2021 (DIPs); findings inform this Special Study.)

4. Decentralization reviews and critiques (both specific to Malawi and more widely. Health and other sectors)

5. Digital health - again both Malawi and more widely

6. Decentralization and gender

7. Documents from other projects

8. Kuunika, decentralization and the Malawian health system (Kuunika project documents)

9. Kuunika, decentralization and the Malawian health system (Kuunika evaluation reports)

10. The Covid-19 pandemic and its impacts on Kuunika

Qualitative data collection was conducted between early September and mid-November 2021

Remote KII were conducted with individuals based in Balaka, Blantyre, Machinga and Zomba Districts. Blantyre and Zomba are Districts that have received support from Kuunika (in November 2018 Zomba received an accelerated core package of project support, specifically on the Demographic Data Exchange and the EMR portal), while Balaka has served throughout the evaluation as a comparator District; Machinga was part of the baseline.

The total sample for Special Study 3 was 54, disaggregated into categories of key informant as set out below. More precise details are not provided, because the Proposal submitted to the National Committee for Science and Technology for ethical approval stated that all participation would be anonymous and confidential, in line with approved evaluation principles. All key informants were given informed consent statements and information about the evaluation in advance of discussion.

All key informants were also asked questions about the project and, therefore, contributed to the endline study.

Table 1: Category of key informant

Limitations include restrictions on travel due to the pandemic and the resultant challenges of conducting remote interviews through Microsoft Teams and Zoom. Connectively problems led to a number of missed key informant interviews (KII). The six Health Facility focus group discussions could only be conducted because Professor Maureen Chirwa travelled to each location and convened the FGDs, which were moderated remotely by Dr Janet Gruber.

In addition, there was a distinct lack of enthusiasm at District level to give time for KII. This was the most challenging cadre to engage with; upwards of 10 additional scheduled KII did not take place, despite repeated attempts by Mott MacDonald and CDM. This was the case in both Kuunika and non-Kuunika Districts.

Another limitation is that due to pandemic restrictions it was not possible to undertake dedicated KII or FGD with Health Surveillance Assistants (HSA)/Disease Control Surveillance Assistants (DCSA) or with any representatives of Health Facility, Village or other category of health committee. This had been possible during the baseline and midline evaluations.

2. Decentralization and health and digital data systems in Malawi

We first of all consider decentralization writ large, i.e. not solely as it pertains to the health sector. We then examine Malawian legislative instruments and processes, again not solely in the context of the health sector. Section 2.3 homes in on digital health and the extent to which national instruments on this topic do or do not address decentralized structures. Finally we look at global digital health principles, guidelines and reports and interrogate these for relative degree of attention to subnational levels.

2.1 Definitions of decentralization

These are not unique to Malawi, but are internationally applied definitions and are, therefore, those used in this report

It may be that deconcentration (see below) is the closest actual fit for Malawi - in which operations are decentralised, but decision-making powers are not devolved.

The term decentralization embraces a variety of concepts.

Decentralization refers to the transfer of authority and responsibility for public functions from the central government to subordinate or quasi-independent government organizations and/or the private sector it is a complex multifaceted concept.

Different types of decentralization should be distinguished because they have different characteristics, policy implications, and conditions for success

Administrative decentralization seeks to redistribute authority, responsibility and financial resources for providing public services among different levels of government. It is the transfer of responsibility for the planning, financing and management of certain public functions from the central government and its agencies to field units of government agencies, subordinate units or levels of government, semiautonomous public authorities or corporations, or area-wide, regional or functional authorities.

There are broadly two major forms of administrative decentralization - deconcentration and devolution

Deconcentration redistributes decision making authority and financial and management responsibilities, usually among different levels of the central government.

The overall consensus is that devolution is a form of administrative decentralization. Thus: devolution is the statutory delegation of powers from the central government of a sovereign state to govern at a subnational level.

The transfer of power and authority may involve revenue generation, priority setting, resource management and/or decision making, and the sub-national units may be elected directly by the population, or appointed by the central level or by private entities. These multiple modes of decentralization make it a very complex concept to study in a real world setting.

A key concept in the decentralization debate is decision space = the degree of discretion that peripheral units have within the law, and their ability to ‘bend the law’, with implications for accountability - very much including the ability of actors to demand from or provide information to others within the system (see e.g. Bossert 1998; Tsofa, Molyneux et al 2017).

Another is the 'matching principle': this is widely seen as a basic (but seldom fully or even partially achieved) requirement for an efficient and effective sub-national government. Thus for any public service, the benefit areas (e.g. a Health Facility catchment area) should be matched by the financing areas (areas over which fees or taxes are being levied to finance the service). Moreover, expenditure responsibilities should be matched with revenue sources and revenue capacities should be matched with political accountability.

2.2 The global digital health context: relative focus on decentralized structures

There has been considerable work carried out in the past two decades in particular to expand and embed the use of digital data systems in the global South. Yet such work frequently lacks attention to decentralized health structures (whether through discussing their potential/actual inputs, possible pitfalls, any evidence for or against such focus, etc), while calling for greater equity of data collection and use, which would presumably require engagement at all levels of a health system, horizontally as well as vertically.

Recent initiatives include the May 2018 71st World Health Assembly Resolution on Digital Health, which demonstrated global recognition of the value of digital technologies to contribute to advancing universal health coverage (UHC) and other health aims of the Sustainable Development Goals (SDGs).

The Resolution urged Ministries of Health to assess their use of digital technologies for health […] and to prioritize, as appropriate, the development, evaluation, implementation, scale-up and greater use of digital technologies, as a means of promoting equitable, affordable and universal access to health for all, including the special needs of groups that are vulnerable in the context of digital health...to consider, as appropriate, how digital technologies could be integrated into existing health systems infrastructures and regulation, to reinforce national and global health priorities by optimizing existing platforms and services (p. 2).

However, the 2018 Resolution did not discuss decentralized aspects of digital health data systems.

Moreover, with respect to the SDGs: none of SDG 3 (Good Health and Wellbeing) targets or indicators makes reference to digital data, indeed to any data, aggregate or disaggregated. SDG 9 and its 9a, 9b and 9c targets tangentially refer to ICT access, more sustainable infrastructure, research capabilities and the like: all relevant to digital health data collection and use, which is nowhere specified.

The 2019 WHO guideline: recommendations on digital interventions for health system strengthening emphasizes the pivotal necessity of digital data for supporting improved health service delivery, while highlighting the challenges ahead

Thus the Guideline states: Amid the heightened interest, digital health has also been characterized by implementation rolled out in the absence of careful examination of the evidence base on benefits and harms. The enthusiasm for digital health has also driven a proliferation of short-lived implementations and an overwhelming diversity of digital tools. (p. i)

Despite the wide-reaching nature of the Guideline, it does not address space for decentralized digital health engagement; the adapted Tanahashi framework model set out in the Guideline fails to include such wider governance and management levels in its overview of opportunities for digital health specific to achieving UHC.

The widely endorsed (including by BMGF) nine Principles for Digital Development, first developed in 2012 3, address issues such as:

Designing with the user: this supports the building of better, more transparent, jointly shared and robust digital data systems designed with the context and user in mind.

Understanding the existing ecosystem: this requires the involvement of 'community members, donors, local and national governments' in an iterative process throughout a project/initiative lifecycle.

3 Principles of Digital Development. Retrieved from: https://digitalprinciples.org/ The Principles are an attempt to unify digital principles and create a community of practice for those who work in digital development. The Digital Principles were first created in consultation with organizations such as BMGF, SIDA, UNICEF, UNDP, the World Bank, USAID and WHO.

Designing for scale: in other words, thinking beyond the pilot and the proliferation of often smallscale, vertical projects that are probably implemented in a limited area - and work from the outset to understand what works.

While not mentioned, this would ideally include attention to the District level; any pilot would build in scale-up strategies from the beginning.

The 2021 World Development Report, entitled Data for Better Lives, does not discuss in any detail the role or remit of decentralized structures (health or other) in the context of building effective, democratic systems with strong governance. It does, however, note the following (NB: without specific reference to decentralized or sub-national structures, unless these are implicitly subsumed under 'government agencies'):

'When government agencies, civil society, academia, and the private sector securely take part in a national data system, the potential uses of data expand and so does the potential impact on development. In fact, the more integrated the system and the more participants involved, the higher is the potential return...Higher degrees of integration require close coordination and shared governance between participants, but such integration is otherwise compatible with a decentralized data architecture...Even though most countries are far away from the aspirational goal of a well-functioning data system, setting sights on this target can provide countries with guidance on the next steps in developing such a system ' (p. 16)

A 2018 report entitled Transforming Health Systems Through Good Digital Health Governance notes that Digital health has been acknowledged as a key building block for UHC and the healthrelated Sustainable Development Goals. However, it argues that while much work is being undertaken across the global South, a holistic approach to digital health, which requires good governance for successful implementation and sustainability throughout the health system, continues to be lacking in many countries

Thus: Good governance is needed at all levels local, district, provincial, and national throughout the health information system. This, in turn, supports equitable access and delivery of quality, affordable health services. (Marcelo et al. 2018; p. 4)

Such efforts to entrench good governance in the Malawian digital health context should surely require engagement with and by District health structures, in order to optimize buy-in, ownership and the development of a 'knowledge culture' 4 for planning and service delivery. Relevant here is that in order to develop a knowledge culture, which is a process-driven incremental build, there needs to be support to such activities as well as training; the evidence throughout the independent evaluation is that training on its own is insufficient to increase appetite for, and use of, digital data for planning and to achieve improved service delivery and health outcomes.

Again, it is important to note that no one project can hope to achieve all such outcomes, because there needs to be a national, systemic process running alongside to address decentralization in its broadest sense.

2.3 A brief overview of Malawi decentralization legislation and processes

The box below homes in on the health sector and the extent to which overarching national legislative instruments and reports and also sector strategies address decentralization and decentralized systems and structures.

4 Here 'knowledge culture' refers to a group of behaviours, including responsiveness towards the use of data, positive evidence of individual and organizational practices to support data use, senior staff members actively promoting data use and encouraging users, all of which can develop and sustain evidence-based planning and service delivery.

Literature review 5

• The 1998 Malawi Local Government Act, the key instrument in terms of legislative foundations for decentralization, refers to devolution.

• Political decentralization stalled in 2005. There has at best been 'piecemeal' fiscal + administrative decentralization since then. The 2010 amendments to the 1998 LGA 'clawed back' many aspects of decentralization (p. vii re. both formal + informal recentralization of power and functions). There is lack of coherence/read across between District Development Plans + District Health Implementation Plans (O' Neill, Cammack et al 2014)

• The 1998 LGA left 'referral Health Facilities' within the ambit of the central MoH, but did not clearly define what is meant by a referral Health Facility The 2010 Local Government (Amendment) Bill undermined decentralization + the management of public sector reform. The major lesson provided by Malawi is that there is no automatic relationship between [partial/incomplete] decentralization + efficient public service delivery + development. (Hussein 2012)

• The 2017 National Health Plan II and accompanying Health Sector Strategic Plan II identify the importance of improved governance + strategies to achieve more effective cooperation with stakeholders. The partial implementation of decentralization is viewed as a cause of poor governance in the health sector.

• The current Malawi Health Sector Strategic Plan II 2017 - 2022 (the HSSP II) includes little specific consideration of decentralization. It is unclear whether there has been any coherence between the development of the HSSP II + iterations of the National Decentralization Programme

• Furthermore, the 2020 Malawi Voluntary National Review Report for the SDGs contains minimal discussion of decentralized/District health structures or their role in work towards achievement of the SDGs. Thus the section on SDG3 contains no mention of Districts' role in its achievement, or of the importance of data disaggregated by District and contextual priorities.

• HSSP III is currently under development - there is said to be impetus for achieving one single data reporting system (rather that the existing proliferation of donor-supported mechanisms in addition to DHIS2). In addition, a 'SWAp revival' concept note is under development. There is apparently genuine appetite to ensure HSSP3 applies a more integrated approach to health service delivery, away from programme silos.

The literature review reveals a range of frequently voiced concerns linked to decentralization in Malawi and elsewhere. The points raised in the quotes are echoed in this report.

'Stakeholders view governance challenges as a significant barrier to achieving a more effective and equitable health system. Three categories were identified: accountability (enforceability; answerability; stakeholder-led initiatives); health resource management (healthcare financing; drug supply); influence in decision-making (unequal power; stakeholder engagement)...The partial implementation of the government policy of decentralization was viewed as a cause of poor governance in the health sector.' (Masefield et al 2020)

'The decentralization agenda remains incomplete, even nascent, in Malawi. Genuine decentralized government requires effective coalitions between central and local state and nonstate actors.' (Mohmand & Loureiro 2017)

The following quotes (the first from a Kuunika partner representative) are more specific to decentralization of health data architecture and systems.

'Disjointed planning and asymmetric access to information... Despite the mandate to create periodic District Development Plans (DDPs) and District Implementation Plans (DIPs) with associated budgets, funding and resource allocation decisions are mostly determined centrally... Most plans go underfunded and tough decisions must be made about what will get prioritized...After submission to national level, plans are often returned with a prescribed budget

5 For details of all references see Annex 1.

reduction and a surprisingly short turn-around time to revise and resubmit leaving little or no room for negotiation backed by specific data. (Smith 2015; pp. 24 - 25)

Lack of attention to gender, equity and other inequalities in Malawian (and other) health system decentralization processes is significant.

'Gender responsive health systems' can be defined as health systems that 'address the gender determinants of health, the gender factors at work in the health system and the resulting gender inequalities'. Achieving gender-responsive health systems requires the integration of gender into decentralization processes and health system governance. ' (Pendleton et al 2015).

The WHO Health Equity Monitor states that: Monitoring health inequalities is essential for achieving health equity. Health inequality monitoring uses health data disaggregated by relevant inequality dimensions ... in order to identify differences in health between different population subgroups. Disaggregated data provide evidence on who is being left behind and informs equityoriented policies, programmes and practices.

2.4 Digital health in Malawi: national instruments and their relative attention to decentralized structures

Literature review

• Key documents relating to digital health are the National Health Information Systems Policy (2015) + the Monitoring, Evaluation and Health Information Systems Strategy (MEHIS), 2017-2022. The 2020-2025 National Digital Health Strategy has recently replaced the 2011-2016 National eHealth Strategy.

• The NHISP: guiding principles include the need to generate locally relevant data; disaggregation by sex, age, geographical areas, income groups so as to 'achieve greater equity, efficiency and quality'; development of data use; holistic approach; robust [e-Health] systems. Stated barriers to effective use of data include: vertical +/or parallel data systems; lack of interoperability; lack of data sharing; too few trained in data management

• MEHIS (not solely e-Health): Reference to lack of co-ordination of MEHIS activities, e.g. multiple teams separately organizing supportive supervision and performance reviews Despite recent efforts to harmonize systems, major weakness of the HIS is existence of vertical/parallel reporting systems. Development partner support concentrates HIS resources in individual programs, ignoring/weakening national HMIS. Despite such discussion, MEHIS does not refer to decentralisation in the context of digital data.

• The NDHS: this comprehensive strategy addresses Districts primarily in terms of resource gaps; ownership + leadership in relation to digital data receive minimal focus. No real discussion of the role of DHIS2; the same is true for decentralization + its ramifications + impacts

• A Kuunika deliverable is development of Standard Operating Procedures 6. MEHIS states 15 are to be completed; 12 are at various stages of development.

Key informant interviews

• Many of the key documents that are supposed to underpin digital data systems are great on paper but weak on implementation - and all too often far too ambitious in terms of indicators. This is especially true at decentralized levels (National level key informant)

• There are now Government and MoH guidelines and strategies, but those tend to leave out the District levelit's as though they are the 'missing middle'. But any national strategy needs to include all levels in planning for use of digital data, otherwise why bother to collect all that information? (National level key informant)

• There are documents, national strategies, the MEHIS - but none of those really thinks about the role of the District, or the Health Facilities. The focus is always on the national level, while most of the health sector work is in the Districts (DHO staff

6 See Annex 3 for full review of the SOPs specific to relative focus on decentralized structures and District role and responsibilities.

o None of the SOPs under development include detailed attention to decentralized health structures + the role of edata within such a system.

o Only two SOPs refer explicitly to Districts' roles + responsibilities: User Support and DQA

o E.g. the SOP Guidelines for the Development and Revision of HIS Standard Operating Procedures (no number) defines what a SOP should be, their relevance. The section on roles + responsibilities does not include District structures, while e.g. development partners are addressed. Districts are not explictly mentioned as part of the 'secondary audience' for the SOPs, while donors, CSOs, universities, etc. are.

o There is no reference in any of the reviewed SOPs to Data Clerks or HMIS Officers, the cadre of health worker who are tasked with data entry.

o SOP 11 - Introduction of new e-Health in the HIS landscape of MoHP Malawi sets out an MoHP defined hierarchy of expertise + 'end users'. Districts + lower levels are in the latter group.

member, non-Kuunika District

• One big thing Kuunika has done is to support the development of the 20202025 National Digital Health Strategy. That's important and a major step. But it could have looked more at the Districts, what we can't do, what we could do and how we get from A to B. (DHO staff member, Kuunika District)

The extent to which Malawi's digital data architecture and systems support effective collection and use of disaggregated data at District level - and why this is important - will be considered in the review of the District Implementation Plans (see 4.3 below).

3. Kuunika: Data for Action

Kuunika is about to begin its Phase 2; the intention of this report is that it will provide utilizationfocused findings, lessons learned and recommendations on actions, processes and next steps, specific to the subject of decentralization as linked to digital health data systems, so as to inform the second phase of the project.

This Special Study will only describe the activities and outputs relevant to the 2018 project pivot and the 2019 sustainability phase insofar as they refer to decentralization. The assumption here is that the readers of this report will have sufficient knowledge of the project to contextualize more broadly; see also the full endline report for detailed discussion.

The Kuunika timeline (copied from the December 2020 project overview) from project inception in early 2017 to the final pivot to support the pandemic response is as follows:

See section 4.2 for discussion of Kuunika inputs to the Covid-19 response.

3.1 Kuunika and decentralization: general points

It is important to emphasize again that decentralization of the health system and digital data architecture and use is beyond the control of any project, including Kuunika Such factors include the long-standing human resources for health challenges, which are especially acute in rural areas, the fact that many Data Clerks, on whose shoulders rest much of the daily task of data entry, receive too little training and other support and are often co-opted from other roles, e.g. as gardeners. Moreover, decisions at national government level as to the extent and detail of any devolution of financial autonomy to Districts and sectors are beyond any project's control.

In addition, attention by the project partners to the specifics of decentralization and the ramifications for Kuunika has throughout been limited. As a result, the independent evaluation of Kuunika has hitherto not examined decentralization issues in much detail.

Therefore, what this Special Study does is to look at the space available to Kuunika in terms of decentralized systems and structures and examines if the project has most effectively engaged with Districts and lower levels to optimize impact of its activities and support.

Despite decentralization as a political and administrative process being beyond the Kuunika mandate, little detailed thought appears to have been given since 2016 by the project to providing optimal opportunity for effective work at District level This would involve planning effectively for how its inputs, staff members and digital data work might most coherently and comprehensively enable decentralized structures to be more engaged partners in a digital health ecosystem

Figure 2: Kuunika Timeline

Thus the Ministry of Health (MoH) has been described as the prime beneficiary and owner of the project throughout Kuunika implementation. This refers chiefly to the central level.

Kuunika has had a somewhat intermittent experience in its focus on, and direct relations with, participating Districts. District HIV and AIDS services and their clients have been intended Kuunika beneficiaries from the project start in early 2017. Capacity development of health workers and District health officials remains one of the three unchanging pillars of the project. Yet the August 2016 Kuunika Proposal and initial Implementation Plan make few explicit references to centralDistrict relations.

It was not until the first ‘pivot’ in Kuunika implementation in 2018 that District Health Offices (DHO) became involved in the project Steering Committee and DHO priorities were taken into account in project plans. That pivot to the 'core package' focused on the development of key capabilities to support data use at national, District, facility and community levels during the remainder of phase 1 of Kuunika

Organizational change at District level has received at best modest attention from Kuunika A significant body of research literature identifies an intermediate stage (or on-going process) of organisational culture change which is necessary to translate more and better data into better decision-making and outcomes. This process of organizational change has not been fully articulated or addressed by Kuunika Organizational change represents a core area where actual decentralization processes, however limited, need to be acknowledged and where the project should ideally have been flexible and responsive to realities on the ground at District level.

Examined through the lens of health system decentralization and District engagement, how should the twists and turns in the progress of Kuunika – primarily a digital HMIS intervention - be regarded? A number of published studies analyze the functional effectiveness of decentralization, in terms of bureaucratic capabilities, using concepts of ‘decision space’, innovation and performance.

Our evaluation work to date suggests that as a digital data systems project, Kuunika might be most usefully regarded as sitting in the ‘functions and capabilities’ space, its contribution (where possible) to decentralization of health services lying in its ability to empower Districts via access to better data to plan and manage - and to provide evidence of improvements in both. This presupposes effective sharing of data, horizontally as well as vertically, and by all partners, all necessary components of genuine aid effectiveness (for consideration of this, see 4.5)

These are issues to be explored in sections 3 and 4 of this study. In 3.2 and 3.3 we provide first of all a brief overview of key points that emerged from the independent Kuunika baseline and midline evaluations. Section 4 discusses endline/Special Study findings.

3.2 Key findings from the 2017 Kuunika baseline evaluation specific to decentralization

These were the most pertinent findings made about the processes and implications of decentralization as linked to the project:

• Lack of initial engagement with Districts: certain members of the original project consortium did not prioritize working with the Districts. The initial focus of Kuunika was on building data systems and not who at which level (other than the national) might have access or indeed ownership. As a result, communication between Districts and national levels was lacking.

• Kuunika was seen as having been designed somewhat in a 'vacuum', without sufficient attention to how best to develop and apply communication, reporting and feedback channels between the funder, the GoM/MoH, project consortium partners and actual data collectors and users.

• Concern over lack of precise information about the Kuunika structures and decision-making channels in support of Districts was apparent.

• The points above are underlined by the view of one 2021 key informant with knowledge of the entirety of project implementation: to start with, Kuunika was almost entirely working at the national level. Significant activities at District level were deployment of EMRs and data quality improvement inputs. This was the situation for the first two years of implementation - and the

cessation of such work by Kuunika represents an opportunity missed

• Of note is that Kuunika and other donor funding (at the time of the baseline, the midline and in 2021) supports many health activities at District level; it was pointed out that such programs, all too often with vertical M&E and reporting systems, may well have greater impact on Districts and Health Facilities than any action on decentralization, especially if that process continues to be significantly underfunded.

• Related to the point immediately above about proliferation of donors, programs and data systems: one significant issue noted by a number of baseline key informants was that certain of the original Kuunika consortium members were also implementing CDC programs, including in the five Kuunika Districts. This led to some confusion, feelings of 'double dipping' in terms of collection and use of data, as well as (reportedly) tension. Such comments were not subsequently made during data collection for either the midline or the endline.

More general comments on decentralization and the health sector included:

• Weariness over the length of time that was being taken over bringing about decentralization of health systems: 'We continue to await the euphoria of decentralization' (District Health Management Team (DHMT) member).

• A number of respondents pointed out that more knowledge was needed on how budgetary and decision-making powers would/could be devolved specific to HIV and health and which government bodies at which level or levels would have the final say.

• Decentralization (to the extent it had happened by 2017) had ushered in more involvement of village and Ward level committees. Zonal, District Health Management Team (DHMT) and District Council respondents described how such community structures often sought to put forward their own health agendas, without any basis in data quality and verification.

• Decentralization might usher in more District ownership of data; however, unless digital and other health data are reliable, such ownership might be somewhat redundant in terms of usefulness in planning and improving service delivery and outcomes.

3.3 Key findings from the 2019 Kuunika midline evaluation specific to decentralization

• Information from midline KII was that there continued to be some reluctance on the part of some project consortium members to engage at District level. While some work was being done to change this viewpoint, much energy was being given to developing the project paper entitled Project Implementation Plan - operationalizing Kuunika sustainability (finished on 31st July 2019). This diverted time and capacity away from District engagement. This was considered to have been a (further) missed opportunity to engage properly with the District level.

• The unlocking of all four 'key capabilities' as set out in the 2019 - 2020 Kuunika sustainability causal pathways cannot genuinely and sustainably be achieved without attention to support at all levels: training, data management, data use and development of a knowledge management organizational culture.

• As at the baseline, the midline found very little use of financial or HRH data by those supported by the project, in part for sure reflecting the limited nature of fiscal decentralization to date.

• The development of the Local Authority MIS (LAMIS) was described as a potentially valuable tool for decentralized ownership of data, not least because information is that it will include a number of databases, not least DHIS2.

• A range of views were expressed on the relevance and usefulness of the DHIS2:

o Respondents on DHMTs and at the District Councils referred to the District Implementation Plans as opportunities for using DHIS2 data to provide tailored and specific evidence to inform planning and budget allocations.

o There continued to be a widespread perception of ‘data gathering for data extraction’.This was a consistent theme at District level. Both EMR and DHIS2 data were perceived to ‘funnel up’ data from facilities and districts to the national level. Such extraction and exclusion were felt to inhibit the use of analysed data for administrative, service delivery or planning purposes, and limit opportunities for the growth of evidence-based planning.

o Less than one in five (18%) of all midline respondents used DHIS2 to look up data. A common concern was that the DHIS2 contains ‘too many indicators’. A majority of District respondents felt that the DHIS2 was ‘one size fits all’ with no scope for districts to tailor it to

include indicators of particular relevance to their work or geographical area.

o Poor quality of data was seen as having repercussions for district level planning, including the ability to argue robustly for budget and human resource allocations The quality of routine data was said to be a fairly regular topic of concern at district level meetings where DHMT, other clinicians and/or District Council staff worked together. There were concerns at national, zonal and district levels over the time lag between data collection and feeding into the DHIS2 system.

• Key to genuinely effective decentralization is engagement with ALL levels of the health system. Therefore, a midline finding of concern was that most Health Facility Advisory Committee members had inadequate knowledge of data collection and use and could not independently determine if data were of good quality. HSAs also described barriers to knowledge and use of data, despite being key partners in data collection for HIV (at the time, the Kuunika use case) and other data. HSAs were not included in any data use decisionmaking at Health Facility level. The majority of HSAs had not been trained on data entry.

One point discussed both at the midline and more coherently during endline data collection by key informants with a longitudinal perspective on the entirety of Kuunika, is that in 2019 a 'convening point', i.e. an organizational center and management hub, did not really exist. The development and strengthening of the MoH Digital Health Division is now seen as an integral component of digital data governance and national ownership. The same key informants were less positive about the extent to which such organizational framework building has so far been expanded down to District level.

As from November 2019 the core package work was extended into phase 2, with expressed increased attention to sustainability (including at District level). District Health Offices (DHOs) were then discussing how Kuunika could best support them to address District-level priorities. At the time it was planned that DHOs were to be more fully and actively involved in the project. Indeed this was a midline finding and a recommendation followed to ensure greater engagement of DHOs.

The 2019 Kuunika Implementation Plan notes that 'Feedback provided by subnational MoH during planning meetings for the final year work schedule and budgets pointed to the lack of full awareness and leadership involvement in implementation of activities. To address this, the Kuunika secretariat also embarked on a consultation progress with the five Kuunika districts to share the activities planned for 2019/2020 and receive feedback on district-level stakeholder priorities' (p. 3)

Of relevance here is that such consultation was to occur at least 2.5 years into project implementation; this begs the question - why not during planning or inception phase? The 2019 midline evaluation amplified such lack of dialogue and effective ownership and that is a feature finding of the endline too. Such incomplete focus on decentralized structures has deep roots, in that the June 2018 Kuunika core package pivot to 'unlock the 5 key capabilities' did not explicitly refer to District engagement, ownership, governance.

Nonetheless, one clear sign of progress at this stage of the project is that towards the end of 2018/early 2019, more District activities were underway, because recruitment of Kuunika officers to be sited at DHOs had been completed. The data use campaign was launched. So, District activities were achieving some momentum In addition, at the time of the midline there were indications of Districts becoming slightly more active partners in Kuunika, with DHO staff members feeling they had space to discuss addressing context-specific priorities.

Post-2019 opportunities specific to entrenching structures and systems for digital data at District level appear to have been missed - or more feasibly, been (partially or wholly) overtaken by the Covid-19 pivot.

4. Special Study 3 findings

4.1 General findings from the 2021 endline

Many 2021 findings echo, indeed amplify, those from the baseline and the midline in terms of lack of project involvement in the development of truly decentralized, less vertical systems of digital (and other) data collection and use. 7 Of course, such governance and often politicized issues are beyond the mandate of any one program. However, a significant number of 2021 key informants at all levels (national, District and Health Facility) see Kuunika as a program that has always primarily supported national digital data structures, while to an extent having previously been more visible and active in Districts. District engagement includes supporting EMR use and maintenance, providing considerable training, undertaking data use campaigns, providing facility dashboards in Zomba, and so on.

Key informant interviews: views on Kuunika's engagement with Districts - looking across its implementation 2017 - 2021

• Kuunika wasn't initially plugged into the Districts re. digital data use, ownership. The incomplete decentralization structures didn't help. Initially the Kuunika consortium was almost a parallel MoH, without any attention to decentralized health + data. (National level key informant)

• The original consortium didn't recognize that Kuunika is a project FOR the MoH + the GoM. Communication between District and national levels was lagging + lacking - Kuunika was somewhat designed in a vacuum, without thinking of communication channels between the project + Districts. The initial big focus in Kuunika was data - not which entity/individuals had access, ownership, etc. Later focus was more on patient outcomes + use of data to support service delivery; that necessitates proper District buy-in if improvements in service delivery are to be achieved. (National level key informant)

• When Kuunika was being designed, the GoM said it was decentralizing the health system, but most decisions + human resource management continued to be made at the central level. There is a centralized 'push system' for sure in health. Most projects at District level have to work very closely with the MoH, even if more focused on the Districts; this is true for Kuunika. (National level key informant)

• For sure Kuunika has provided significant training, hardware (e.g. initially EMR support; latterly e.g. facility dashboards), technical support, etc, but it has not sought to systematize such inputs, i.e. to help to build people's understanding of why it is so important to analyze and use data. (DHO staff member, Kuunika District)

• At Health Facility level, Kuunika is often seen as distant and disengaged, before and during the Covid-19 response. (DHO staff member, Kuunika District)

One key Special Study 3 finding is that the cumulative experience of the Kuunika evaluation shows that there needs to be greater genuine ownership of data at the District level. This necessitates buy-in from all partners working in any one District and nationally, challenging in a donor landscape currently as fragmented as Malawi; this has obvious implications for effective coordination of digital health initiatives. The District Health Offices should ideally be integral partners in data collection and use.

In addition:

Development of a knowledge culture, where individuals, institutions and organizations genuinely acknowledge and internalize the understanding that data collection and use will improve health

7 Review of the 2019 Vital Wave report entitled Assessment of EMR Systems in Malawi. Prepared for the MoH, Republic of Malawi. Initial Landscape Assessment indicates many similarities in its areas of focus and those of this special study (e.g. lack of knowledge culture; lack of support to Data Clerks; lack of attention to demand-side/ equity issues). One interesting difference is the absence in the Vital Wave report of any detailed consideration of District-level data structures and systems.

planning and service delivery, and where stakeholders work together effectively, is hampered by what is perceived to be the lack of performance/results-based management in the Malawian public sector (including health). Kuunika has not been able to make much headway in supporting acknowledgement of the need for, or the institutionalization of, such a knowledge culture.

Information is that there are currently (from mid-2021 onwards) changes being made at national government level specific to what might be seen as a (re) centralization of digital health data management and (re) focus on DHIS2. For instance, Digital Health has been moved back into the MoH Department of Planning and Policy Development.

There is opinion that this might dilute a developing center of excellence in terms of trained personnel. The move might limit, even reduce, Districts' and health workers' capacity to review and make speedy use of data; such moves might also have impacts on the building of digital data systems and knowledge culture at District and facility levels

The project's community engagement is also considered to have reduced over time, apart from those few key informants at DHOs who discussed OHSP and IDSR work during the pandemic and how the community component has been important for gathering key surveillance data.

So can we define the constraining and supporting factors shaping Kuunika's engagement at District level and below?

Factor analysis

• Limited space for decentralization of the Malawian health system

• Limited financial decentralization + budgetary control

• Limited attention to District health structures in Kuunika Proposal and initial design, despite (?) there being Malawian consortium partners

Constraining Factors

• Kuunika has provided very considerable training on DHIS2 across the project Districts, but limited integration of data actually collected into District planning

• DHIS2, supposed to be primarily a District-level digital data system, has limited user base at DHOs + in Health Facilities, and also at community level

• The view prevails that data collected at District + lower levels, by whichever partner, are primarily extracted in a vertical process, with insufficient discussion or joint planning

• Evidence of slowly expanding appetite for development of a knowledge culture in District health systems through use of digital data + DHIS2, supported by the project

Supporting Factors

Looking forward

o To an extent supported since 2017 by Kuunika and other projects' inputs, e.g. training of Data Clerks, appointment of Kuunika HMIS Officers at DHOs, EMRs, dashboards, WhatsApp, Cluster meetings, DIP engagement

o Data 'super users' are pathfinders + their engagement has been prompted/ expanded through Kuunika

o Engagement with HSAs, CBOs, communities + to a lesser extent health Committees primarily positive but relatively limited in terms of outputs

• Engagement with District Councils, under whose remit come the District Health Offices + DHMT, + support through Kuunika-funded staff membersthose staff have facilitated access to digital data for a wide range of District officers, whose direct involvement with the data remains limited; Council officers especially often view DHO DHIS2 data as 'challenging'

One big lesson is that a project cannot just design digital data systems (or expand those) without properly thinking of who (up and down) will use them, how they will be enabled to use them and what the capacity development needs will be.

If it were possible for Kuunika phase 2 to make an input into the development of a knowledge culture at District level, this would surely enhance opportunities for greater and more effective use of digital data in planning and service delivery.

The Blantyre Prevention Strategy (funded by BMGF and on which Cooper/Smith is a partner) is said to have been designed based on lessons learned about Kuunika gaps. This new program has the DHO as the lead, from the start. 'We rethought the process, based on Kuunika challenges regarding decentralization'.

Therefore, while there must be attention to limiting the further proliferation of digital data pilot projects, the BPS may have lessons that can be scaled up to other Districts.

4.2 Kuunika, the Covid-19 pandemic and the Districts

Once the pandemic hit Malawi, project attention and energy focused on developing the C-19 digital data response. Thus in March 2020 there was a further Kuunika pivot, answering the Ministry of Health’s request for assistance with national data collection and monitoring of the Covid-19 pandemic. This appears to have shifted Kuunika’s focus back to the center As yet there is no documentation on whether and how such work might inform future Kuunika District-level engagement and support.

One quite consistent view from both internal project and external national key informants is that Kuunika was catalytic in supporting the Ministry of Health to deploy and pivot to collect Covid-19 data. This has all been done using open source systems and is seen as an example of the importance of the interoperability layer.

Information from Cooper/Smith is that: Covid shifted priorities and the timeline of activities, but the re-evaluation of remaining activities did not flag a permanent change. This may be supported by the fact that our role in some of these activities has shifted from development to technical guidance and oversight.

In addition and specific to whether Covid-19 response inputs from Kuunika have relevant longer term lessons: there have been a lot of lessons learned in coordinating the scale up of a national tool that supports different segments of a system e.g. surveillance, clinical and laboratory services. [There are also] lessons on user support, partner engagement, data quality, data access, and governance.

This graphic is copied from the June 2020 Kuunika document Sustainability Phase & Covid-19 Pivot: overview and sets out key areas of support.

Figure 3:

document Sustainability Phase & Covid-19 Pivot

Kuunika

The updating and rollout of the One Health Surveillance Platform (the OHSP, repurposed from the existing Integrated Disease Surveillance and Response (IDSR) system) represents a major component of inputs provided to and working with Districts (including Health Facilities and communities)

Experiences of its implementation are varied among DHO and Health Facility key informants. Relevant here is that whatever individuals' perspectives on the national Covid-19 response and the actions taken to collect data, all those with any project-specific information felt that Kuunika had sought to make speedy, flexible and responsive inputs in a very challenging environment and was to be commended for such support

Key informant interviews: the OHSP, Covid-19 data collection + use

• In April 2020, Kuunika deployed some 30 phones to IDSR co-ordinators + oriented them on capturing Covid data. There was orientation of Data Clerks on how best to capture the data. Some of the data sets on the OHSP allowed us to gather information on cases, which were entered on the system e.g. contact registration + follow up - Kuunika helped us with tracing of C-19 through contact registration and follow-up, working with the HSAs; OHSP captured data, working with IDSR Focal Persons in the communities. Kuunika also helped us with clinical case management, because data on hospital admissions could be captured from time to time. I think Kuunika made some inputs to orienting data people at each facility? (DHO staff member, Kuunika District)

• Kuunika came to see us collecting data for Covid-19, from the Health Facilities. I haven't yet had any feedback on the data collected or what they mean for us in the pandemic. There is one Data Clerk here helping in the data collection for Kuunika (DHO staff member, non-Kuunika District)

• Data collection has been a very major challenge. I'm not sure what Kuunika has done to help with this (or what they've tried to do, because the situation continues to be bad). Service delivery on all fronts (i.e. not just for Covid) has been delayed, so data collection has been compromised. Generation of data and reports have been delayed, incomplete, weak. Vaccination data for C-19: so much information has not yet been uploaded. Data should be electronically entered, but this was delayed, there were the usual network problems, etc, so we resorted to manual data entry. There is a huge backlog. (DHO staff member, Kuunika District)

• We have had to use the existing structures, the IDSR Unit and its staff. There was no co-ordination, so efficacy was limited. Each partner comes with its own ideas and budget; there should be co-ordination at the MoH. (DHO staff member, Kuunika District)

• There has been quite some activity. There is a person called the District Data something under Kuunika [this is the Data Use Facilitator] and there has been a lot of training of HMIS Clerks and IDSR staff to collect Covid-19 data for Kuunika. (DHO staff member, Kuunika District)

• We haven't had any training here on Covid-19 data entry or use. But the DHO has given us some information, through WhatsApp. (Health Facility staff member, non-Kuunika District)

• We have had no extra support to collect Covid-19 data, from Kuunika or any other project. It would have been good to have that support, e.g. on how to collect Covid-19 data, plus specific indicators and a checklist. (Health Facility staff member, Kuunika District)

• There hasn't really been any support from the District or any project during Covid in terms of data collection, other than the DHO giving us order and inventory forms. There's been no support on data analysis and use and review meetings have been cancelled (Health Facility staff member, Kuunika District)

Review of 2020 and 2021 Kuunika District Quarterly Support Reports, training reports and District Activity Plans reveals the following:

• Very considerable inputs have been made by the project in terms of supporting data collection in Districts, through training of IDSR staff, Data Clerks and others

• The post of Data Use Facilitator has been created and one such person has been working in each of the Kuunika-supported Districts

• Detailed Covid-19 data have been collected

• Despite such support, considerable data backlogs have occurred, due to volume of data to be entered on to OHSP (which some IDSR Focal Persons find hard to use) and lack of time for many Data Clerks. This means that major exercises in data cleaning and back data entry have had to take place, e.g. in Lilongwe

• Such backlogs will of course have implications for completeness and overall quality of Covid-19 data

The Kuunika District Support Quarterly Report (for Q2 2021) states: Blantyre district accumulated a huge backlog of 31,094 COVID-19 vaccination data [as of 10th April] due to increased workload and inadequate devices to capture data. This affected the overall quality of e-vaccination data.

• Despite the considerable inputs documented in Kuunika reports, accurate knowledge of its work specific to Covid-19 response does not appear to be widespread at District or Health Facility level (and information at second hand is that the same is true in many instances at community level)

• For some District and Health Facility key informants, the Covid-19 digital data intervention appears as another vertical, data extraction exercise, whose main focus is not to support optimal and equitable service delivery or to facilitate increased access to and use of evidence.

4.3 One very practical example: has Kuunika been involved with the development of District (health) Implementation Plans?

This is a topic that receives a range of significantly differing responses. We have been prompted to consider it here because of key informants' sharply differing views of Kuunika inputs to the DIP development process. As elsewhere, the necessary caveat is that no single project can be expected to achieve all necessary change.

The evidence indicates that the project has engaged with DIP development in Zomba District; elsewhere key informants are less certain as to any involvement

One striking finding is that there is minimal discussion of any aspect of DIP in national health or digital data for health strategies. This absence does not provide an ideal foundation for systematizing evidence-based planning at any level, but of course particularly in Districts, Health Facilities and communities

Literature review

• New annual (2022) District (health) Implementation Plans (DIP) are currently being developed

• Clinton Health Access Initiative (CHAI) is a major partner in support of DIP development, as is UNICEF

• Few national strategies even refer to DIP development or implementation processes; therefore, presumably none views these as core components of effective planning + service delivery

Key informant interviews

• 'Before 2020, DIPs were based on each program co-ordinator's degree of commitment. Districts now have a very comprehensive spreadsheet, they do bottleneck analysis. The DIP get budgetedsaying what is GoM/ partner/not funded. However, one major challenge is that the Treasury does not disburse what is requested, plus it is always a struggle to harmonize funding coming through partners, i.e. to get them to co-ordinate. We are seeking to get those District partners involved in DIP development. ' (National level key informant)

o The sole reference in MEHIS 2017 - 2022 is discussion of work towards the achievement of the HIS Strategy 2011 - 2016 Objective 8, where HIS data are stated as having been used during DIP development

o NDHS: no reference

o The undated (but 2016 or later) MoH Data and Digital Priorities: Digital Health for Universal Health Coverage is similarly silent

• No information has been forthcoming as to whether the HSSP III, currently in development, will address any aspect of DIP

• Neither the Blantyre Prevention Strategy Condensed Strategy document nor the BPS Fact Sheet refers to the DIP as a planning tool using digital (DHIS2 + other) data, nor addresses the extent to which either the 2021 or the 2022 DIP does or does not make use of DHIS2 data + Bottleneck Analysis to inform service delivery planning + implementation

• 'Kuunika has helped, directly, to develop the new DIP, helping to capture important aspects of data, in terms of team engagement too.' (DHO staff member, Kuunika District)

• 'The DIP is renewed on an annual basis. DHIS2 has become increasingly useful, in terms of developing targets + indicators; each co-ordinator is expected to draft indicators for his/her area. There isn't yet comprehensive use of DHIS2 data (annualised) to address gaps, etc, for the next annual plan. We use DHIS2 as the databank. DIP are not publicly available; they're working documents. We don't have enough funds to disseminate the DIP to the communities, which is a shame + means Committees + CBOs don't get to be part of the debate. Kuunika involvement? Not sure if there is any - certainly not here, but in other Districts?' (DHO staff member, nonKuunika District)

Key question 1: do any DIP show evidence of DHIS2 data supporting evidence-based planning and/or better evidence use? Are Bottleneck Analysis (BNA) findings actually applied to planning for improvements in health service delivery?

One national key informant made the following points, not specific to Kuunika, but revealing about the current status of quality data availability.

'At Health Facility level data quality is a major, major issue. We have sought to use evidencebased planning and bottleneck analysis looking at both the demand and supply sides. The UNICEF Tanahashi Framework is used by one program to analyse the bottlenecks - but in Malawi the data are so lacking or so weak, the categories frequently don't make sense. Quite often the data are reported at above 100% or below 0%. The DIP process reveals that so many data sets just don't add up, in any way. In almost every District the data are so poor, it's more lip service on evidence-based planning.'

The last point is true also for Districts supported by Kuunika, according to the key informant.

Another national level key informant, also with experience of District-level data capacity, stated:

'Some data don't make sense when bottleneck analysis is done - if data are above 100%, then quality is so very poor. The DHIS2 has so many gaps, for indicators and also across time periods - so completeness of data is lacking as well as quality.'

Key question 2: do Kuunika-supported Districts provide any indications of greater engagement with evidence-based planning?

There are some indications of this. Thus the Q2 2021 Kuunika District Support Quarterly Report indicates that the Data Use Facilitator for Zomba (a post funded by Kuunika) participated in a threeday DIP consolidation meeting...supported by CHAI. The main activity was to consolidate all the programme activities into the main DIP template and fill in the performance of different indicators.

Factor Analysis of the 2019-2020, 2020-2021 and 2021-2022 Zomba DIP (District Planning and Budgeting Tool) in terms of use of DHIS2 data + Bottleneck Analysis (BNA)

Constraining Factors

• 2019-2020 DIP does not include BNA

• Lack of clarity as to why the DHIS2 indicators included in the Excel spreadsheet on BNA (PS2) for 2020-2021 + 2021-2022 DIP have been chosen

• Many District programs + their indicators are not (yet) on DHIS2

• It is unclear whether there is any automated link between DHIS2 + PS2 + the BNA

• 2021-2022 PS2 spreadsheet shows that only circa 50% of the DHIS2 indicators listed have a national target

• Many indicators in the 2020-2021 and 2021-202 DIP state 'no activity logged against this indicator'

• Most of the DHIS2 indicators on PS2 are not in the BNA Application, making for limited analysis + read across

• The 2021-2022 DIP lists more indicators performing 'equal or below' the national target measures than above

• It is not possible to gauge the quality of the data included in the DIP

• Issues of disaggregated data or an equity focus appear not to be part of DIP planning (or implementation?) - are these essential levels of complexity too much to ask for at present?

• It is unclear whether the BNA has informed inclusion of new DIP activities

• The extent to which the DIP has been implemented year on year is unclear, yet tracking that is key

• Compared to the 2019-2020 DIP, the subsequent 2 years' Plans indicate more engagement with data + more contextualizing of data to address Zomba priorities

• The BNA uses DHIS2 data + informs identification of root causes

Supporting Factors

• The fact that the DIP is now populated using charts based on DHIS2 + other data is positiveagain, no facility to gauge quality of the data used

4.4 Kuunika evaluation hypotheses relevant to decentralization: findings from Special Study 3

Here we track the entirety of the evaluation work from 2017 with regard to the hypotheses that underpin the Theory of Change and the originally five, now seven, evaluation questions. Five of the 21 hypotheses have particular relevance for this Special Study.

The degree of organisational and political decentralization can affect use of evidence in decision making (hypothesis 5)

2017 (pre-baseline planning + baseline): weak and conflicting KAP shows some culture in data use, uptake and challenge (probably reflecting large % of clinicians in the sample), also increasing at higher levels. The qualitative research shows that zonal and district decision makers (both generalists and health sector) are unsure of the overall parameters of decentralization e.g. will the zonal health teams continue to exist? Will the generalists without health expertise and also the many 'new' levels, e.g. village and Ward committees, have more sway in deciding annual foci for health?

2019 (midline): It has been difficult to gauge the extent to which the ongoing decentralisation (a process that has been ongoing for twenty years or more) is influencing the use of evidence in decision-making. The few District level respondents available to the midline did mention increased use of DHIS2 to develop District Implementation Plans; its expansion was considered potentially positive for creation of increasingly detailed District plans. The development of the Local Authority MIS system was described as a potentially valuable tool for decentralised ownership of data. As at the baseline, the midline found very little use of financial or Human Resources for Health data, reflecting the limited nature of fiscal decentralisation to date

2021 endline and Special Study 3: The hypothesis is assumed to be true, but has never been tested. The Malawi health sector is de jure quite decentralized, de facto not so much. Still to find evidence of greater use of data in DIPs, etc. How to break this circle? Covid-19 monitoring experience has possibly reinforced the sense of a centralized, push system - and perhaps led to greater acceptance of this by some at District level? The pandemic does not appear to have led to more democratization/localization of data capture and use Districts continue to be out of the loop for use of evidence for decision making - and to an extent in its actual use.

Policy making is often messy and opportunistic - ‘a disorderly set of interconnections and back and forthness' (links primarily into hypotheses 7 and 17)

Hypothesis 7

2017 (pre-baseline + baseline): There is more than 1 route by which better data influence policy and practice – by feeding into higher level (especially, for HIV, international) policy making and by informing local planning and decision-making.

2019 (ML): the formal/international policy making level dominates in HIV in Malawi. Need to dig more into ‘informal’ and messy decision making in the districts in the special studies? Pretty messy for sure, given decentralisation (or more precisely, its absence, despite 'formal' enactment) and also external partners' influence.

2021 endline and Special Study 3: governance issues are relevant here. Data collection indicated that analysis and modelling of Covid-19 data by Kuunika showcased the power of data amongst central planners and excited a lot of debate.

Hypothesis 17

2017 (pre-baseline + baseline): by limiting access to the improved DHIS2 data, the role of other groups (outside the formal health system) that help accelerate change may be restricted e.g. the District Policy Planning Officers, who don't know if they'll have such access, but seem to be increasingly important pivot points in the decentralised structure.

2019 (midline): the key pivot point here in terms of developing a knowledge culture that supports policy/ practice effectiveness is for individuals and institutions to realise that effective data use and evidence-based policy making are likely to result in enhanced service delivery, better health outcomes and ultimately support clinical health workers, health planners and data entry personnel in their work. In other words, how to change 'institutional norms' and make engagement with data use a truly positive attitudinal and behavioural action.

2021 endline and Special Study 3: has such 'back and forthness' continued under the weight of pandemic response? Has the pandemic speeded up any policy/governance actions and/or shifts? Are there signs of greater/lesser order and coherence in policy development processes, not least given the urgency of dealing with C-19 in 2020?

Hierarchical management of information, organisational silos can limit access to data and its use. Divisions of responsibilities and ‘silos’ can also limit consideration of evidence (hypothesis 15)

2017 (pre-baseline + baseline): widely/more accessible (within the health system) data on DHIS2 may break down these hierarchies

Depends though on who has user rights and relatively easy access to the system. The 2017 baseline found very little use of DHIS2 at facilities and extensive views that data collection is extractive. There were few obvious feedback loops. There were clear ‘job silos’ - Data Clerks enter data but do not analyse; in-charges (supposedly) analyse but do not enter.

2019 (midline): while the health system is undoubtedly hierarchical, not least in terms of positioning data collection at the lowest level of a steep pyramid, that structure does not result in lack of awareness of the important work done by Data Clerks, or any demonstrated superiority from those with clinical and/or management training and experience. The Kuunika-led cluster meetings and QI sessions which we found at the midline to be well accepted could be interpreted as working by breaking down hierarchies and job silos around data use. By limiting access to the improved DHIS2 data, the role of other groups (outside the formal health system) that help accelerate change may be restricted.

2021 endline and Special Study 3: it appears that Kuunika has not really expanded focus specific to (more? less?) hierarchical management of data. There are indications that Covid-19 response/data collection and use has become a silo and for some District and lower level key

informants has represented challenging demands.

Individuals are empowered through access to data (hypothesis 18)

2017 (pre-baseline + baseline): DHIS2 will be more effective if it is designed with the users’ needs in mind

2019 (midline): limited evidence of such shifts (empowerment is interpreted in this context as meaning personal job satisfaction, acknowledgement that access to data improves the quality of one's work and opportunities for advancement). No evidence has been found of individuals (as compared to facilities) being awarded DHA Certificates, receiving commendations or other professional satisfaction because they have increased access to data.

There was expressed concern about achieving an appropriate balance between greater access to routine health data and what might be required of individual health workers. Empowerment might come with attached demands for increased data productivity, more analysis, etc., so whose empowerment would it actually be?

'Super data users' very clearly understand and celebrate the opportunities given by greater collection and analysis of routine health data and are empowered personally (this includes the one Data Clerk who is such a user, who has led data quality improvement activities at Health Facility level, with expressed support from all clinical colleagues). So too do policy makers at national level, whose position in the health hierarchy is already empowered.

As at baseline: DHIS2 will be more effective if designed with users' needs in mind. The midline found that Kuunika steps to develop DHIS2 using user-driven design approaches appears to be working to promote interest and anticipation as to what can be done + in connecting with a few self-propelled 'super (data) users'.

2021 endline and Special Study 3: what might have changed as a result of the pandemic - any greater horizontal empowerment? The pandemic has resulted in (temporary?) cessation of WhatsApp groups and review meetings. Access to DHIS2 does not seem to have greatly expanded. How might the slowly expanding group of 'super users' become more integrated into District digital data systems (without placing unreasonable demands on them)?

5. How has Kuunika performed against aid effectiveness principles?

In this section of the report we apply a broader perspective, to consider Kuunika's inputs throughout the life of the project and the extent to which its inputs, intermediate outcomes and outcomes may have contributed to aid effectiveness. The 2005 Paris Declaration foregrounded the central importance of aid effectiveness principles being agreed, adopted and applied.

Reflections on Kuunika’s experience in the Districts against the benchmark of aid effectiveness principles

Aid effectiveness principle

1.

OWNERSHIP:

Developing countries should be owners of their development

Assessment of Kuunika’s implementation experience

● Kuunika has throughout worked with the central MoH, which is the owner of the project. However, where should national ownership be sited? Only at the central level, or also at District and lower levels, where the great majority of health service delivery takes place?

● There is considerably less ownership in the Districts, where throughout the lifetime of the project issues have been raised of data extraction, multiple (and sometimes incompatible) reporting systems, vertical programs reporting to the national level, donor agendas sometimes outweighing District priorities in terms of data collection and use.

● Kuunika's position at central level has meant that its inputs have been aligned to MoH and by extension, GoM, policies and systems.

● Kuunika's support to the development of the 2020-2025 National Digital Health Strategy reflects project commitment to alignment (but see e.g. section 2.4 for discussion of this Strategy)

● Major project technical, financial and human resource support to the development of the MoH Digital Health Division is another major benchmark of adherence to the principle of national alignment

2. ALIGNMENT: Development assistance should be aligned to country policies, institutions and local systems

● So too is the development of the Covid-19 digital data response, where Kuunika, other development partners and the GoM (primarily but not solely the MoH through the Digital Health Division) have worked together

● But such achievements are less supportive of true alignment between central and lower levels of the health system, digital data collection and use at District, Health Facility and community levels.

● This begs the question: if Malawi has a highly centralized health system, including for digital data, what is the most effective role of any one intervention such as Kuunika in seeking to expand the definition of alignment, when there is evidence that in 2021 Districts are not yet full partners in the digital data ecosystem?

● Furthermore: aid effectiveness in the context of alignment specific to digital data systems should surely include focus on optimal alignment of sub-national and national structures - Kuunika has not yet optimally supported such work?

3. HARMONISATION: Developing countries and partners should

● Evidence indicates that a '[Health] Sector-wide Approach (SWAp) Revival' Concept Note is in development, an activity supported by the GoM + a number of partners; this must signal a degree of harmonization intent to

Aid effectiveness principle

Assessment of Kuunika’s implementation experience

harmonise their action address the current stated 'extreme fragmentation' of donor partner inputs to health

● Kuunika support to the development and staffing of the MoH Digital Health Division represents a springboard for greater GoM/donor partner harmonization specific to digital health; more effort is required to realize such potential

● Potentially: Kuunika may have generated added value that has enabled other partners to support digital data, e.g. the Global Fund.

4. MANAGING FOR RESULTS: Developing countries and donors should focus on measurable results

5. MUTUAL ACCOUNTABILITY

Developing countries and their partners are jointly accountable for development results

● Digital data collection, analysis and use remains weak at District and lower levels, despite major inputs from Kuunika and other partners: evidence-based results management is nascent at best

● Knowledge culture: optimizing results in the digital data landscape requires genuine and sustained buy-in by individuals and institutions to the principle that data matter and their use can and should improve planning and service delivery. If Kuunika were able to support all health system levels to develop a results-centered data use culture, this could have substantial impact on performance

● This continues to be poor. As at the baseline, so too in 2021: data are vertically extracted by both MoH + donor partner projects from Districts + Health Facilities, with too little discussion, feedback, wider dissemination

● Therefore, Districts and below appear to be the 'missing middle' in terms of accountability (in both respects, i.e. holding others to account + being held accountable - although for the latter, as ever Data Clerks bear the brunt of data entry accountability)

Of note here is that the aid effectiveness principles have been criticized for gender and equity blindness, as having a certain 'one size fits all' approach to such matters.

In the case of digital data and Kuunika (and indeed all other partners working in this area in Malawi), just one point is disaggregation of data - their collection, their identification, their analysis and their use. WHO, UN Women, many civil society organizations and others continue to press for greater equity of data disaggregation, greater acknowledgement that data are never neutral, and greater application of such principles in the context of digital data.

6. Synthesis of Key Findings

In summary, key lessons include:

Decentralization, Districts & Kuunika

• Neither a decentralized 'decision space' for health planning nor functions + capabilities have been effectively supported

• Development of digital health guidelines + systems should be supported more consistently at the District level

• That support should be based on a more coherent approach to the development of an organizational knowledge culture: starting with the users, not the system

• The Covid-19 pivot has not resulted in clear wins for DHOs + Health Facilities (+ perhaps communities): primarily seen as data extraction without effective support.

Sustainability

• Kuunika's key partner is the MoH; however, sustainability of digital health systems requires support to Districts and lower levels, as the central point of service deliveryfor HIV as well as universal health coverage

• The project support to the development of the MoH Digital Health Division represents not only a solid + (it is hoped) sustainable gain; the DHD could also serve as a central point for more aligned + harmonized MoH + partner engagement in work on digital health data systems + use, e.g. if the 'SWAp revival' concept note results in more coordinated inputs

• Kuunika inputs to e.g. the 2020-2025 National Digital Health Strategy represent normative + longer term contributions that can underpin more sustainable digital data architecture

Aid effectiveness

• There continues to be data extraction, imposition of donor partner data systems + indicators, lack of DHO + Council engagement in planning + creation of context-specific plans.

• There is scope (+ urgency) for greater coherence + genuine partnership at District level in digital data collection + use.

• There is fragmentation of effort, due to lack of donor co-ordination across the many health programs and projects. At District level, DHO, Council and Health Facility staff are overwhelmed by multiple demands and reporting formats. This affects the efficacy and efficiency of DHIS2. Alignment of all partners, with the active engagement of District actors, is essential.

• Districts continue to be the 'missing middle' in terms of the digital data architecture

Key findings

1. Decentralization of the health system and digital data is beyond the control of any project, including Kuunika. Information is that there are currently (mid-2021 onwards) changes being made at national government level specific to what might be seen as a (re) centralization of digital health data management and (re) focus on DHIS2. Thus the Digital Health Division has been moved back into the MoH Department of Planning and Policy Department.

2. The move might limit, even reduce, districts' and health workers' capacity to review and make speedy use of data; such moves might also have impacts on the building of digital data

systems and knowledge culture at District and facility levels.

3. The partial, piecemeal and stop-start implementation of the government policy of decentralization is viewed as a cause of poor governance in the health sector Governance challenges are a significant barrier to achieving a more effective and equitable health system in three key domains: accountability (enforceability; answerability; stakeholder-led initiatives); health resource management (healthcare financing; drug supply); influence in decision-making (unequal power; stakeholder engagement).

4. Districts are the 'missing middle' in many respects in terms of GoM and donor partner engagement with the health sector; there is increased focus on community engagement (and quite rightly, not least for equity considerations), yet Districts are the entry point to the great bulk of health services delivered to Malawians.

5. When Kuunika was being designed, the GoM said it was decentralizing the health system, but most decisions and human resource management continued to be made at the central level. There is a centralized 'push system' in health. Most projects at District level have had to work very closely with the MoH, even if more focused on the Districts; this has been true for Kuunika.

6. The MoH is the owner of Kuunika - this has been made clear from project inception. Therefore, has the project made best use of the space available within the predominantly centralized health systems and structure to engage with Districts to build sustainable, standalone use of digital data? The answer is on balance no.

7. The cumulative experience of the Kuunika evaluation shows that there needs to be greater genuine ownership of data at the District level. This necessitates buy-in from all partners working in any one District and nationally, challenging in a donor landscape currently as fragmented as Malawi; this has obvious implications for effective coordination of digital health initiatives. The District Health Offices should ideally be integral partners in data collection and use

8 Kuunika has invested a very great deal of time, effort and resources into District level capacity development, training on and access to DHIS2 and digital data hardware and systems (e.g. dashboards, the mobile App, Cluster meetings). This was most apparent before 2019, at which point changes in consortium partners and project management, coupled with the sustainability pivot, led to Kuunika being seen as becoming more distant from the Districts (except in Zomba) More recent support to horizontal engagement, e.g. through helping to facilitate setting up Cluster meetings, are often not recognized as having Kuunika inputs, but are seen by those involved as useful channels for debate and decision-making processes.

9. The Covid-19 pivot, centrally facilitated by the MoH with Kuunika and other partner support, has enabled very considerable progress to be made by Malawi in terms of tracking the pandemic and developing a response. At District level there are mixed views as to how much such data collection has supported service delivery.

10. Despite being defined in terms of being a District-focus programme in the early days, Kuunika has given relatively limited attention to planning effectively for how it might most coherently and comprehensively support decentralized structures and systems and provide optimal inputs at District level, for access to and use of digital data, through the DHIS2 platform. The temporary involvement of the Districts in Kuunika planning after the first pivot appears to have been short-term and never properly integrated in project planning and processes

11. Kuunika might be most usefully regarded as sitting in the ‘functions and capabilities’ space, its contribution to decentralization of health services lying in its ability to empower Districts via access to better data to plan, manage and deliver services

12. The decision to focus on HIV as the data use case, a vertical system owned at the central level, (and where key datasets collected and managed by the DNHA were not uploaded to DHIS2) meant there was little space for the District level. Initially the Kuunika consortium was almost a parallel MoH, without any attention to decentralized health and data. Kuunika moved also; at one time it was in the HIV Department.

13 Kuunika was somewhat designed in a vacuum, without thinking of communication channels between the project and Districts. The initial big focus in Kuunika was data - not which entity/individuals had access, ownership, etc. Later focus was more on patient outcomes and use of data to support service delivery. That necessitates proper District buy-in if improvements in service delivery are to be achieved - and that step has not been properly taken by the project

14. At the end of Kuunika phase 1, there is little concrete evidence of sustained, systemic improvements in data use knowledge culture at District level, or of key planning and service delivery documents such as DIP being progressively informed by quality data derived from DHIS2. There continue to be considerable problems of access to DHIS2, of capacity to navigate its programs, of identifying, analysing and using quality data.

15 The existence of data 'super users' demonstrate the possibilities for building a knowledge culture based on interest twinned with capacity. The question is how to maximize such potential without thereby placing unrealistic burdens on individuals.

16 The Blantyre Prevention Strategy is said to be designed based on lessons learned regarding Kuunika District gaps. This new program has the DHO as the lead, from the start. 'We re-thought the process, based on Kuunika challenges with decentralization.'

7. Recommendations for Kuunika phase 2

Recommendation 1: engage throughout with Districts - leadership, ownership and governance

Rationale: the ultimate goal of any data system should be to deliver optimal health services that lead to improved patient outcomes. That necessitates proper District buy-in, which was not embedded into Kuunika from the outset.

Recommendation 2: work for greater aid effectiveness - alignment with other partners working on digital health data

Rationale: information from national level respondents is that a 'Sector-wide Approach (SWAp) Revival' Concept Note is in development. There is increasing emphasis on maturing away from a proliferation of pilot projects and toward proven and scaled solutions built on common standards within an architecture.

The consensus appears to be that any such action would seek to re-introduce SWAp principles to alignment, joint planning, working and Monitoring, Evaluation and Learning (MEL), but not (at least initially) any financial disbursements to government entities. Investments and implementation for digital health data require far more harmonization. Application of a number of SWAp principles could potentially link disbursements to District performance, monitored using digital data. This would, however, necessitate genuinely effective support and training to all those engaged with collecting and using data to plan and deliver services.

Kuunika (and the Bill and Melinda Gates Foundation) should continue work with the MoH to support greater alignment and harmonization of all partners' digital health data interventions. If there is a SWAp-light framework developed for digital health data, Kuunika should consider a role as an integral partner, calling upon cumulative project experience, expertise and its position at the center of national developments.

Recommendation 3:

continue to support the MoH Digital Health Division

Rationale: the Digital Data Division should continue to receive Kuunika support; it is the site of government technical capacity. In its inception phase Kuunika 2 should plan for greater DDH coordination with and support to District digital health data systems.

Recommendation 4: ensure digital data systems are designed with users in mind and work to maximise all levels having access to participation, training and ongoing support

Rationale: systems should not be designed without properly thinking of who will use them and how, and the capacity development each individual cadre will need. With hindsight, the adoption of HIV as a data use case for Kuunika may have limited scope and flexibility - collection of HIV data was and remains tightly managed at the national level. Districts and Health Facilities did not have oversight of such data, or effective ownership.

Recommendation 5:

build in sustainability from the outset

Rationale: while there is increasing resistance to a 'proliferation of pilots' in the digital health data sphere, Kuunika 2 has options to build on the foundations not only of the project, but to engage closely with the Blantyre Prevention Strategy, which is focused on building Districts systems, capacity and ownership. Such relationships, allied to any development of greater partner alignment and harmonization, could enable not only economies of scale but evidence-based prioritization of interventions proven to be effective.

Recommendation 6: have more focus on equity aspects of digital data systems

Rationale: data are never neutral. Just one point is disaggregation of data - their collection, their identification, their analysis and their use. WHO, UN Women, many civil society organizations and

others continue to press for greater equity of data disaggregation, and greater application of such principles in the context of digital data.

Annex 1: Literature Review References

1. Definitions + debates: decentralisation, devolution, deconcentration Conyers D (2017). 'Foreword'. IDS Bulletin Vol. 48 (2); pp. vii-x. (Issue entitled 'Interrogating Decentralisation in Africa')

Hussein M (2012). 'Decentralisation and Management Reforms on the Death Bed? Obstacles Facing Malawi's District Councils.' Africa Review 4:1; pp. 33-47. http://dx.doi.org/10.1080/09744053.2013.764119 (NB: maybe originally 2005?)

Mohmand SK & M Loureiro (2017). 'Introduction: Interrogating Decentralisation in Africa'. IDS Bulletin Vol. 48 (2); pp. 1 - 14. (Issue entitled 'Interrogating Decentralisation in Africa')

Slack E, Spicer Z & M Montacer (2014). Decentralization and Gender Equity. Forum of Federations: Global Network on Federalism and Devolved Government. Occasional Paper Series # 14.

Yuliani E (2004). Decentralization, deconcentration and devolution: what do they mean?

2. Government of Malawi Documents, e.g. Strategies, NDPs

Government of Malawi (GoM). The Health Sector Strategic Plan 2017 - 2022

GoM (2015). National HIS Policy (NHISP).

GoM (2017). Malawi Growth and Development Strategy (MGDS) III

GoM (2018). Monitoring, Evaluation and Health Information Systems Strategy 2017 - 2022 (MEHIS)

GoM (2020b). National Digital Health Strategy 2020 - 20205 (NDHS)

GoM (2020b). Voluntary National Review Report for the SDGs: Main Report.

UNDP + GoM( 2014). Review of the 2nd National Decentralisation Strategy (NDS II)

3. District Development Plans, etc.

Balaka DDP 2017 - 2022.

Blantyre DDP 2017 - 2022.

Machinga DDP 2017 - 2022

Machinga Socio-economic Profile (SEP) 2017 - 2022

Zomba DDP 2017 - 2022.

District (health) Implementation Plans

Ministry of Health (2019a). Balaka District Planning and Budgeting Tool (Financial Year 2019-2020).

Lilongwe: MoH Department of Planning and Policy Development.

MoH (2019b) Blantyre District Planning and Budgeting Tool (FY 2019-2020). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2019c) Machinga District Planning and Budgeting Tool (FY 2019-2020). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2019d). Zomba District Planning and Budgeting Tool (FY 2019-2020). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2020a). Balaka District Planning and Budgeting Tool (FY 2020-2021). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2020b). Blantyre District Planning and Budgeting Tool (FY 2020-2021). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2020c). Machinga District Planning and Budgeting Tool (FY 2020-2021). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2020d). Zomba District Planning and Budgeting Tool (FY 2020-2021). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2021a). Balaka District Planning and Budgeting Tool (FY 2021-2022). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2021b). Blantyre District Planning and Budgeting Tool (FY 2021-2022). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2021c). Machinga District Planning and Budgeting Tool (FY 2021-2022). Lilongwe: MoH

Department of Planning and Policy Development.

MoH (2021d). Zomba District Planning and Budgeting Tool (FY 2021-2022). Lilongwe: MoH Department of Planning and Policy Development.

4a. Decentralisation/overall governance dox: reviews and critiques - Malawi (specific both to health + wider)

Hara M (2008). 'Dilemmas of Democratic Decentralisation in Mangochi District, Malawi: Interest and Mistrust in Fisheries Management.' Conservation and Society 6 (1); pp. 74 - 86. Hussein M (2012). 'Decentralisation and Management Reforms on the Death Bed? Obstacles Facing Malawi's District Councils.' Africa Review 4:1; pp. 33-47

http://dx.doi.org/10.1080/09744053.2013.764119 [NB: maybe originally 2005?]

Jagero N, Kwandayi HH & A Longwe (2014). 'Challenges of Decentralization in Malawi.' International Journal of Management Sciences 2 (7); pp. 315-322

https://www.academia.edu/6891568/Challenges_of_Decentralization_in_Malawi

Masefield S, Msosa A & J Grugel (2020). 'Challenges to effective governance in a low income healthcare system: a qualitative study of stakeholder perceptions in Malawi.' BMC Health Services Research (20): 1142. https://doi.org/10.1186/s12913-020-06002-x

O'Neill T, Cammack D et al (2014). Fragmented governance and local service delivery in Malawi

London: Overseas Development Institute https://cdn.odi.org/media/documents/8943.pdf

4b. Decentralisation/overall governance dox: reviews and critiques - NOT Malawi (specific both to health + wider)

Bossert T (1998). Analysing the decentralisation of health systems in developing countries: decision space, innovation and performance. Soc Sci Med: 47: 1; pp. 47 – 60. DOI: 10.1016/s0277-9536(98)00234-2

Dehnavieh R, Haghdoost A, Khosravi A et al. (2019). 'The District Health Information System (DHIS2): A literature review and metasynthesis of its strengths and operational challenges based on the experiences of 11 countries'. Health Information Management Journal 48 (2): pp. 62 - 75. DOI: 10.1177/1833358318777713

Malakoane B, Heunis J, Chikobvu P et al (2020). 'Public health system challenges in the Free State, South Africa: a situation appraisal to inform health system strengthening'. BMC Health Services Research (2020) 20: 58 doi.org/10.1186/s12913-019-4862-y

McCollum R, Limato R, Otiso L et al. (2018). 'Health system governance following devolution: comparing experiences of decentralisation in Kenya and Indonesia.' BMJ Glob Health 3:e000939. doi:10.1136/bmjgh-2018- 000939

Tsofa B, Molyneux S et al. (2017). 'How does decentralisation affect health sector planning and financial management? a case study of early effects of devolution in Kilifi County, Kenya.'

International Journal for Equity in Health

16:151 DOI

10.1186/s12939-017-0649-0

Zon H, Pavlova M, Drabo KM & W Groot (2017). 'Municipal health services provision by local governments: a systematic review of experiences in decentralized Sub-Saharan African countries.' Health Policy Plan; 32 (9):pp. 1327 -1336. DOI: 10.1093/heapol/czx082

5a. Digital Health (+ other) management/governance: Malawi (incl. GoM dox)

Chikumba A (2017). 'Management of Health Information in Malawi: Role of Technology.' Advances in Science, Technology and Engineering Systems Journal: Vol. 2 (1); pp. 157-166.

GoM and Ministry of Health (NB: dated 2014). Malawi National E-Health Strategy 2011 - 2016

GoM and MoHP (NB: dated 2018). Monitoring, Evaluation + Health Information Systems Strategy [MEHIS] 2017 - 2022

Malik T (2020). Malawi's Journey Towards Transformation - lessons from its national ID project Center for Global Development paper.

Ministry of Health (undated, but 2016 or later). Data and Digital Priorities: Digital Health for Universal Health Coverage

MoHP (2017a). Standard Operating Procedure 01: Revision of indicators and data collection tools: (SOP 01)

MoHP (2017b). SOP: Guidelines to privacy, security and service continuity of HIS in Malawi. (no SOP number)

MoHP (2017c). SOP: User Account Management. (no SOP number)

MoHP (2018a). SOP: Interoperability of HIS. (no SOP number)

MoHP (2018b). SOP: DQA. (no SOP number)

MoHP (2018c). SOP: Data Access and Release (no SOP number)

MoHP (2018d). SOP: Guidelines for the Management of the Master HF Registry. (no SOP number)

MoHP (2018e). SOP: Guidelines for the Development and Revision of HIS Standard Operating Procedures. (no SOP number)

MoHP (2018f). SOP: User Support. (no SOP number)

MoHP (2019) SOP 11: Introduction of new e-Health in the HIS landscape of MoHP Malawi

Smith T (2015). Achieving a Unified System for M&E of the Health Sector in Malawi. Principle [sic] Barriers and Opportunities for Investment.

Vital Wave (2019). Assessment of EMR Systems in Malawi. Prepared for the MoH, Republic of Malawi. Initial Landscape Assessment

5b. Digital Health (+ other) management/governance: NOT Malawi

de Freitas C, Amorim M, Machado H et al. (2021) 'Public and patient involvement in health data governance (DATAGov): protocol of a people-centred, mixed-methods study on data use and sharing for rare diseases care and research.' BMJ Open 11:e044289. doi:10.1136/bmjopen-2020-044289

Germann S, Jasper U (2020). 'Realising the benefits of data driven digitalisation without ignoring the risks: health data governance for health and human rights.' mHealth 2020;6:34. doi: 10.21037/mhealth-2019-di-11

Ibrahim H, Liu X, Zariffa N et al (2021). 'Health data poverty: an assailable barrier to equitable digital health care.' Lancet Digit Health 2021 https://doi.org/10.1016/ S2589-7500(20)30317-4

Kickbusch I, Agrawal A, Jack A, et al. (2019). 'Governing health futures 2030: growing up in a digital world- a joint The Lancet and Financial Times Commission.' The Lancet 394:1309//

Marcelo A, Medeiros D et al (2018). Transforming Health Systems Through Good Digital Health Governance. Asian Development Bank; ADB Sustainable Development Working Paper Series 51. N, George A, LeFevre A (2019). 'How to use relevant data for maximal benefit with minimal risk: digital health data governance to protect vulnerable populations in low-income and middle-income countries.' BMJ Glob Health: 4:e001395. doi:10.1136/bmjgh-2019-001395

Tran Ngoc C, Bigirimana N, Muneene D et al (2018). 'Conclusions of the digital health hub of the Transform Africa Summit (2018): strong government leadership and public-private partnerships are key prerequisites for sustainable scale up of digital health in Africa.' From Digital Health Hub of the Transform Africa Summit (TAS) 2018 Kigali, Rwanda. 8-9 May 2018 BMC Proceedings 12 (Suppl. 11): 17. https://doi.org/10.1186/s12919-018-0156-3

World Bank (2021). World Development Report 2021: Data for Better Lives Washington DC: IBRD. WHO (2018). 71st World Health Assembly Agenda Item 12.4: digital health. 26th May 2018. WHO (2019). WHO guideline: recommendations on digital interventions for health system strengthening. Geneva: World Health Organization (WHO/RHR/19.8).

Plus: Principles of Digital Development. Retrieved from: https://digitalprinciples.org/

6. Decentralisation and Gender

Beall J (2005). Decentralizing Government and Centralizing Gender in Southern Africa: Lessons from the South African Experience Geneva: UNRISD. Occasional Paper 8.

Masanyiwa Z, Niehof A & C Termeer (2014). 'Gender perspectives on decentralisation and service users’ participation in rural Tanzania'. The Journal of Modern African Studies: 52 (1); pp. 95 - 122 doi:10.1017/S0022278X13000815

Pendleton J, Irani L, Mellish M, Mbuya-Brown R & N Yinger (2015). Promoting Gender-Responsive Health Governance: Lessons and Next Steps. Washington, DC: Futures Group, Health Policy Project. Slack E, Spicer Z & M Montacer (2014). Decentralization and Gender Equity. Forum of Federations: Global Network on Federalism and Devolved Government. Occasional Paper Series # 14.

7. Documents from other projects/programmes

Chirwa M & A Behrendt (2021). Spotlight Mid-term Assessment Report using ROM review GiZ (2018). Malawi: HSS with a focus on RH. Malawi-German Health Programme.

Mott MacDonald | Independent evaluation of Kuunika: strengthening HIV related health data systems

VENRO (Association of German Development NGOs) (2010). Local Power and Women's Rights –Gender Perspectives on Decentralisation Processes. Workshop of African and European Civil Society Organisations (November 2009).

8. Kuunika and C/S documents

C/S (2020). Investment Overview, Successes, Lessons, and Thoughts for the Future December 2020.

Kuunika (2019). Project Implementation Plan – Operationalizing Kuunika Sustainability 2019-2020; Kuunika Project Implementation Plan for the Financial Year 2019-2020

Kuunika (2020). Sustainability Phase & COVID19 Pivot. Overview June 8, 2020.

Kuunika (2020) Sustainability phase alignment. [NB: 082020]

Plus a number of 2020 and 2021 Kuunika District Activity Plans and Reports across all five supported Districts.

9. Kuunika evaluation documents

2017. Kuunika Evaluation Baseline Report

2019. Kuunika Evaluation Mid Line Report.

10. The C-19 pandemic + its impacts

Heidari S & H Doyle (2020). 'An invitation to a feminist approach to global health data'. Health and Human Rights Journal (22) 2; pp. 75 - 78.

Stover J, Kelly S, Mudimu E, Green D, T Smith et al (2021). 'The Risks and Benefits of Providing HIV Services during the COVID-19 Pandemic'.

Annex 2: SS3 key informant interview questionnaires

1. Ministry of Health & Population KII guide

1. Has the MoHP seen any changes in the past 5 years in terms of decentralization processes?

2. Has administrative and political decentralization specific to health systems had any impacts on MoHP use of evidence in decision making?

3. Has administrative and political decentralization specific to health had any impacts on MoHP access + inputs to + use of digital health data + data systems in decision making?

4. (How) has the Kuunika project addressed the MoHP's specific digital data needs and challenges

5. Has Kuunika supported Districts in management of digital health data?

6. To what extent have Kuunika outputs supported the MoHP to facilitate the creation of capabilities for digital health data use at the district level?

7. Has Kuunika been agile in identifying + responding to decentralization activities that may have an impact on project work?

2a. What if any have been the changes in terms of health decision-making + service delivery from the perspective of the MoHP?

2b.What if any have been the changes in terms of health decision-making + service delivery at District and sub-District levels, from the perspective of the MoHP?

3a. What if any have been the changes in terms of use of digital data in health decision making at the MoHP?

3b. What if any have been the changes in terms of health decision-making + service delivery at District and sub-District levels, seen from the perspective of the MoHP?

4a. (How) has Kuunika involved the MoHP in project planning (including its pivots)?

4b. (How) has Kuunika involved the MoHP in project delivery?

4c. (How) has Kuunika supported MoHP capacity development in terms of digital health data systems + use?

5a. What has Kuunika done to facilitate MoHP support to Districts' use of DHIS2?

5b. What has Kuunika done to facilitate MoHP support to Districts' use of other digital health data systems?

From the perspective of the MoHP + in terms of optimizing comparative advantage, coherence + effectiveness:

6a. Has Kuunika supported the setting of performance and quality targets?

6b. Has Kuunika provided assistance in routinely measuring performance against targets?

6c. Has Kuunika been responsive to Districts' + Health Facilities' digital health data technical + other challenges?

From the perspective of the MoHP + in terms of optimizing comparative advantage, coherence + effectiveness: (How) has Kuunika contributed to the centre supporting the Districts?

Have Kuunika inputs + outcomes in any way promoted/supported decentralization of health systems + digital data?

Have any of Kuunika’s intended outcomes at the District level been undermined by any tensions/ delays/changes in the decentralization agenda?

8. Does Kuunika’s 2020 ‘Covid pivot’ of Kuunika represent a 'recentralization' of health sector

Does the pivot have potential for generating unanticipated benefits for central-District/local relations and interactions?

resources and decisionmaking?

What has the pivot meant specific to management of + access to digital data for the MoHP overall, for the QMU specifically (+ any other closely involved departments/ divisions/units)?

2. District Health Office, Council and DHMT KII guide

1. Has your District seen any changes in the past 5 years in terms of decentralization processes?

2. Has administrative and political decentralization specific to health systems had any impacts on your (personal + District's) use of evidence in decision making?

3. Has administrative and political decentralization specific to health had any impacts on your (personal + District's) access + inputs to + use of digital health data + data systems in decision making?

4. (How) has the Kuunika project addressed Districts’ specific digital data needs and challenges

5. Has Kuunika supported Districts in management of digital health data?

6. To what extent have Kuunika outputs been adopted and ‘capabilities created’ for digital health data use at the District level?

7. Does Kuunika’s 2020 ‘Covid pivot’ of Kuunika represent a 'recentralization' of health sector resources and decisionmaking?

1a. If there have been changes, will these have any impacts on the development of the new DDPs and DIPs?

2a. What if any have been the changes in terms of health decision making at District and sub-District levels?

2b.What if any have been the changes in terms of health service delivery at District and sub-District levels?

3a. What if any have been the changes in terms of health decision making at District and sub-District levels?

3b. What if any have been the changes in terms of health service delivery at District and sub-District levels?

4a. (How) has Kuunika involved Districts in project planning (including its pivots)?

4b. (How) has Kuunika involved Districts in project delivery?

4c. (How) has Kuunika supported District capacity development in terms of digital health data systems + use?

5a. What has Kuunika done to support the District's use of DHIS2?

5b. What has Kuunika done to support the District's use of other digital health data systems?

6a. Has Kuunika supported the setting of performance and quality targets?

6b. Has Kuunika provided assistance in routinely measuring performance against targets?

6c. Has Kuunika been responsive to Districts' + Health Facilities' digital health data technical + other challenges?

What impacts has the pivot had on Kuunika's support to the Districts?

3. Kuunika and Cooper/Smith KII guide

1. Has Kuunika and/or C/S seen any changes in the past 5 years in terms of decentralization processes?

2. Has Kuunika and/or C/S seen any administrative and political decentralization having any

2a. What if any have been the changes in terms of health decision making at District and sub-District levels?

impacts on use of evidence in decision making?

3. Has administrative and political decentralization specific to health had any impacts on Kuunika and/or C/S supporting Districts' access + inputs to + use of digital health data + data systems in decision making?

4. (How) has Kuunika addressed Districts’ specific digital data needs and challenges ?

5. (How) has Kuunika supported Districts in management of digital health data?

6. To what extent have Kuunika outputs been adopted and ‘capabilities created’ for digital health data use at the District level?

7. Has Kuunika been agile in identifying + responding to decentralization activities that may have an impact on project work?

2b.What if any have been the changes in terms of health service delivery at District and sub-District levels?

3a. What if any have been the changes in terms of health decision making at District and sub-District levels?

3b. What if any have been the changes in terms of health service delivery at District and sub-District levels?

4a. (How) has Kuunika involved Districts in project planning (including its pivots)?

4b. (How) has Kuunika involved Districts in project delivery?

4c. (How) has Kuunika supported District capacity development in terms of digital health data systems + use?

5a. What specifically has Kuunika done to support Districts' use of DHIS2?

5b. What specifically has Kuunika done to support Districts' use of other digital health data systems?

Has Kuunika collaborated with other programmes/projects working on digital health data?

6a. Has Kuunika supported the setting of performance and quality targets? How specifically?

6b. Has Kuunika provided assistance in routinely measuring performance against targets? How specifically?

6c. Has Kuunika been responsive to Districts' + Health Facilities' digital health data technical + other challenges? How specifically?

7a. (How) has Kuunika contributed to the centre supporting the Districts?

7b. Have Kuunika inputs + outcomes in any way promoted/ supported decentralization of health systems + digital data?

7c. Have any of Kuunika’s intended outcomes at the District level been undermined by any tensions/delays/changes in the decentralization agenda?

8. Does Kuunika’s 2020 ‘Covid pivot’ represent a 'recentralization' of health sector resources and decisionmaking?

8a. Does the pivot have potential for generating unanticipated benefits for central-District/local relations and interactions?

8b. What impacts has the pivot had on Kuunika's support to the Districts?

4. KII guide: other programmes/projects

1. Has your organization seen any changes in the past 5 years in terms of decentralization processes?

2. (How) has your programme/ project addressed Districts’

2a. (How) has your programme/project involved Districts in project planning (including its pivots)?

2b. (How) has Kuunika involved Districts in project delivery?

specific digital data needs and challenges?

3. Has your programme/project supported Districts in management of digital health data?

4. Has your programme/project managed to be agile in identifying + responding to decentralization activities that may have an impact on project work?

2c. (How) has Kuunika supported District capacity development in terms of digital health data systems + use?

3a. What has it done to support the District's use of DHIS2?

3b. What has it done to support the District's use of other digital health data systems?

Has your programme/project collaborated in any way with Kuunika?

4a. (How) has your programme/project contributed to the centre supporting the Districts?

4b. Have your programme/project's inputs + outcomes in any way promoted/supported decentralization of health systems + digital data?

4c. Have any of your programme/project’s intended outcomes at the District level been undermined by any tensions/ delays/ changes in the decentralization agenda?

5. Has your programme/project changed its focus in order to respond to the pandemic?

5a. Does any such change of focus have potential for generating unanticipated benefits for central-District/local relations and interactions?

In addition, a small number of KII with national programme partners and international experts on decentralization focused on the following questions.

1. The autonomy and authority of Districts in the context of decentralisation

2. Degree of appetite for genuine decentralisation

3. Challenges, achievements in the context of decentralisation of health systems

4. The current status of decentralisation in the context of health service financing, governance, data collection and use and service delivery.

5. The role of digital data: how data can be/are used at District and national levels to plan and improve health service delivery

6. Any developments over the past 5 years and any projected changes

7. The role and importance of projects such as Kuunika in this context

8. Any changes due to the GoM and MoH response to the pandemic, specific to digital data and the role and remit of Districts

Annex 3: Review of Standard Operating Procedures specific to focus on decentralization and Districts

This review is based on draft SOPs received by the evaluation team. Our understanding is that finalization of the SOPs is work in progress. Our analysis here indicates further work should be considered specific to the extent to which SOPs address Districts' and Health Facilities' roles and responsibilities in the context of digital data.

1. No number: User Account Management - July 2017

Comments specific to decentralization and the roles and responsibilities of Districts: this seems very much to be a 1st draft. Data custodians include District Health Officers; the list of custodians is led by CMED.

2. SOP 01: Revision of indicators and data collection tools - September 2017

Data custodians are defined as primarily departments/units within the MoH

Comments specific to decentralization and the roles and responsibilities of Districts: this SOP does not mention of disaggregation. There is reference to (but no explanation of) SMART indicators

There is no reference to Districts' roles and responsibilities under the remit of this SOP [unless medical staff at District level are subsumed within MoH staff - but presumably CMED and the Digital Health Division will have different priorities to a DHMT

3. No number: Guidelines to privacy, security and service continuity of HIS in MalawiSeptember 2017

This SOP discusses the shared responsibility for ensuring patients' (digital) data are safeguarded. The list includes Zones, Districts and Health Facilities.

Comments specific to decentralization and the roles and responsibilities of Districts: this SOP does not set out governance structures for ensuring confidentiality; it does not consider how each level of the health system might be supported to ensure best practice in this respect.

4. No number: Guidelines for the Development and Revision of HIS Standard Operating Procedures - May 2018

'While acknowledging that there is a multiplicity of stakeholders engaged in the development of SOPs, the primary focus of this document is the MOH&P, especially those people in the HIS domain. The MOH&P is ultimately the key beneficiary in using information generated from HIS and any efforts to strengthen systems should be targeted at the Ministry.

The secondary audience for these guidelines include other relevant stakeholders supporting HIS such as:

• Development partners, donors and NGOs who provide technical assistance, governance, oversight or financing to health programmes and HIS

• Universities and capacity builders, as HIS capacity building efforts are institutionalized at universities and colleges'

Comments specific to decentralization and the roles and responsibilities of Districts: this is the meta-SOP, i.e. it defines what an HIS SOP should be and why SOPs are relevant

Specified roles and responsibilities do not include Districts, but e.g. development partners are listedperhaps this is a true reflection of where/how power is situated?

Does the MoH entirely subsume and control Districts' health systems, structures, facilities? If not, then those levels are not explicitly mentioned in this meta-SOP.

5. No number: Data Access and Release - July 2018

This SOP is stated as being guided by the E-Transactions and Cyber Security Act (2016)

Comments specific to decentralization and the roles and responsibilities of Districts: Districts are not on the list discussing roles and responsibilities; CMED is to lead. Might Districts be assumed to act as 'data custodians'?

This SOP includes considerable discussion on undergraduate access, while not referring at all to e.g. the DH Office, the DHMT, District Council Planners, etc.

Of the ten reviewed, this SOP most addresses governance, oversight, clear lines of sight for accountability and confidentiality of data use - for all data users. However, the discussion is limited. For instance, there is nothing on client/patient informed consent - their data will apparently be made available to even undergraduates. There is nothing about whether, how, any client has any say in the matter, or indeed what the roles and responsibilities of a DHO or Health Facilities might be in this regard.

6. No number: Guidelines for the Management of the Master Health Facility Registry - July 2018

This SOP has the necessary intention of harmonising the various different lists under the aegis/auspices of...CMED.

Comments specific to decentralization and the roles and responsibilities of Districts: DHMTs are listed as one of the key stakeholders in terms of managing the Master Health Facility Register, as custodians of the District's health data. No roles and responsibilities are specified. The DHO is to be responsible for digitizing and uploading annual updates.

7. No number: User Support - July 2018

This is a 1st/even zero draft.

HMIS Officers play their part in supporting HMIS users at District level, working with the DEHO and responsible for systems maintenance, thereby helping e.g. Data Clerks and Programme Co-ordinators.

Comments specific to decentralization and the roles and responsibilities of Districts: This SOP refers specifically to decentralization, thus: 'HMIS Office or IT Support at district level - Support Level I is the district level support guided by the HMIS Officers in collaboration with the IT Support staff. In the spirit of decentralization, all issues encountered at district level should as much as possible be resolved at that level.'

There is no indication of how e.g. human resources for health (or budgets) are to be mobilized and managed at District level to accommodate such activities.

8. No number: Interoperability of HIS - July 2018

Definitions: Interoperability: Interoperability describes the extent to which systems and devices can share data and information across different platforms using HIS established standards, such as LOINC, HL7, RxNorm, etc.

QMD: This is the Quality Management Directorate of MOHP Malawi. It is the division vested with the design, implementation, management and coordination of digital health solutions of MOHP, Malawi (people, processes and technology), working in close collaboration with the ICT Department and the Central Monitoring and Evaluation Division of MOHP.

CMED: The Central Monitoring and Evaluation Division of MOH Malawi is the division vested with the responsibility to coordinate the collection, consolidation, analysis and dissemination of MOH data in line with section 2 of the HIS Policy.

Comments specific to decentralization and the roles and responsibilities of Districts: this is a SOP written entirely from the central perspective of the MoH. No attention is given to how lower levels of the health system are to engage with interoperability, or how these levels might be supported/encouraged to make optimal use of the health planning and decision-making opportunities that would be afforded by effective use of the interoperable DHIS2.

9. No number: DQA - July 2018

Data collected by many levels; the SOP states these should be entered into DHIS2 and be accessible there.

These are the specified roles and responsibilities of DHMTs: 'The DHMT led by the DHO are primarily responsible for providing the necessary support to HMIS officers and all other staff involved in data management and data quality assurance in their respective districts. Using the DQA Module, Districts should conduct monthly quality checks: On monthly basis perform a district-level desk-based review of all key data sets using the integrated DHIS2 data quality tool. File the relevant reports and identify and document the facilities that need data management supportive supervision and mentorship.'

District Quarterly Data Quality Assessment: the quarterly data quality assessment shall be performed by HMIS Officers and program coordinators. The assessment shall build on the desk review process and focus on measuring data accuracy.

Comments specific to decentralization and the roles and responsibilities of Districts: no indication is given of any support to be provided to DHMTs and DHOs (or indeed Health Facilities) in terms of conducting and reporting on either the monthly DHIS2 audits or the quarterly DQAs.

10. SOP # 11: Introduction of new e-Health in the HIS landscape of MoHP Malawi - February 2019

NB: the e-Health sub-TWG Group includes no representatives from District level - all are from central level, mostly from the MoHP.

'End users: These are persons who will use the intended HIS solution. End users include but are not limited to departments, district health offices, district hospitals, health centers, community settings, health development partners, other institutions and ministries beyond MoHP.'

Comments specific to decentralization and the roles and responsibilities of Districts: End users are right at the bottom of an MoHP heavy governance structure for this SOP on e-Health.

There is reference to a pilot phase: [during this] phase, solution provider disseminates as much information on the pilot as possible to all key stakeholders, including but not limited to QMD, CMED, IT Section, relevant Departments, pertinent TWGs and end users.

The hierarchy appears explicit and fails to specify District and Health Facility levels.

Further comments

There is minimal attention in any of the ten reviewed SOPs to any aspects of decentralized/devolved health systems and structures, in terms of roles and responsibilities or access to resources. There is no consideration of how digital health SOPs might facilitate greater ownership of data for planning and decision-making by relevant District and sub-District health actors.

There is equally minimal attention in any of the SOPs to disaggregation of data collection and analysis and why such focus might be useful.