3. Kuunika: Data for Action

Kuunika is about to begin its Phase 2; the intention of this report is that it will provide utilizationfocused findings, lessons learned and recommendations on actions, processes and next steps, specific to the subject of decentralization as linked to digital health data systems, so as to inform the second phase of the project.

This Special Study will only describe the activities and outputs relevant to the 2018 project pivot and the 2019 sustainability phase insofar as they refer to decentralization. The assumption here is that the readers of this report will have sufficient knowledge of the project to contextualize more broadly; see also the full endline report for detailed discussion.

The Kuunika timeline (copied from the December 2020 project overview) from project inception in early 2017 to the final pivot to support the pandemic response is as follows:

See section 4.2 for discussion of Kuunika inputs to the Covid-19 response.

3.1 Kuunika and decentralization: general points

It is important to emphasize again that decentralization of the health system and digital data architecture and use is beyond the control of any project, including Kuunika Such factors include the long-standing human resources for health challenges, which are especially acute in rural areas, the fact that many Data Clerks, on whose shoulders rest much of the daily task of data entry, receive too little training and other support and are often co-opted from other roles, e.g. as gardeners. Moreover, decisions at national government level as to the extent and detail of any devolution of financial autonomy to Districts and sectors are beyond any project's control.

In addition, attention by the project partners to the specifics of decentralization and the ramifications for Kuunika has throughout been limited. As a result, the independent evaluation of Kuunika has hitherto not examined decentralization issues in much detail.

Therefore, what this Special Study does is to look at the space available to Kuunika in terms of decentralized systems and structures and examines if the project has most effectively engaged with Districts and lower levels to optimize impact of its activities and support.

Despite decentralization as a political and administrative process being beyond the Kuunika mandate, little detailed thought appears to have been given since 2016 by the project to providing optimal opportunity for effective work at District level This would involve planning effectively for how its inputs, staff members and digital data work might most coherently and comprehensively enable decentralized structures to be more engaged partners in a digital health ecosystem

Thus the Ministry of Health (MoH) has been described as the prime beneficiary and owner of the project throughout Kuunika implementation. This refers chiefly to the central level.

Kuunika has had a somewhat intermittent experience in its focus on, and direct relations with, participating Districts. District HIV and AIDS services and their clients have been intended Kuunika beneficiaries from the project start in early 2017. Capacity development of health workers and District health officials remains one of the three unchanging pillars of the project. Yet the August 2016 Kuunika Proposal and initial Implementation Plan make few explicit references to centralDistrict relations.

It was not until the first ‘pivot’ in Kuunika implementation in 2018 that District Health Offices (DHO) became involved in the project Steering Committee and DHO priorities were taken into account in project plans. That pivot to the 'core package' focused on the development of key capabilities to support data use at national, District, facility and community levels during the remainder of phase 1 of Kuunika

Organizational change at District level has received at best modest attention from Kuunika A significant body of research literature identifies an intermediate stage (or on-going process) of organisational culture change which is necessary to translate more and better data into better decision-making and outcomes. This process of organizational change has not been fully articulated or addressed by Kuunika Organizational change represents a core area where actual decentralization processes, however limited, need to be acknowledged and where the project should ideally have been flexible and responsive to realities on the ground at District level.

Examined through the lens of health system decentralization and District engagement, how should the twists and turns in the progress of Kuunika – primarily a digital HMIS intervention - be regarded? A number of published studies analyze the functional effectiveness of decentralization, in terms of bureaucratic capabilities, using concepts of ‘decision space’, innovation and performance.

Our evaluation work to date suggests that as a digital data systems project, Kuunika might be most usefully regarded as sitting in the ‘functions and capabilities’ space, its contribution (where possible) to decentralization of health services lying in its ability to empower Districts via access to better data to plan and manage - and to provide evidence of improvements in both. This presupposes effective sharing of data, horizontally as well as vertically, and by all partners, all necessary components of genuine aid effectiveness (for consideration of this, see 4.5)

These are issues to be explored in sections 3 and 4 of this study. In 3.2 and 3.3 we provide first of all a brief overview of key points that emerged from the independent Kuunika baseline and midline evaluations. Section 4 discusses endline/Special Study findings.

3.2 Key findings from the 2017 Kuunika baseline evaluation specific to decentralization

These were the most pertinent findings made about the processes and implications of decentralization as linked to the project:

• Lack of initial engagement with Districts: certain members of the original project consortium did not prioritize working with the Districts. The initial focus of Kuunika was on building data systems and not who at which level (other than the national) might have access or indeed ownership. As a result, communication between Districts and national levels was lacking.

• Kuunika was seen as having been designed somewhat in a 'vacuum', without sufficient attention to how best to develop and apply communication, reporting and feedback channels between the funder, the GoM/MoH, project consortium partners and actual data collectors and users.

• Concern over lack of precise information about the Kuunika structures and decision-making channels in support of Districts was apparent.

• The points above are underlined by the view of one 2021 key informant with knowledge of the entirety of project implementation: to start with, Kuunika was almost entirely working at the national level. Significant activities at District level were deployment of EMRs and data quality improvement inputs. This was the situation for the first two years of implementation - and the cessation of such work by Kuunika represents an opportunity missed

• Of note is that Kuunika and other donor funding (at the time of the baseline, the midline and in 2021) supports many health activities at District level; it was pointed out that such programs, all too often with vertical M&E and reporting systems, may well have greater impact on Districts and Health Facilities than any action on decentralization, especially if that process continues to be significantly underfunded.

• Related to the point immediately above about proliferation of donors, programs and data systems: one significant issue noted by a number of baseline key informants was that certain of the original Kuunika consortium members were also implementing CDC programs, including in the five Kuunika Districts. This led to some confusion, feelings of 'double dipping' in terms of collection and use of data, as well as (reportedly) tension. Such comments were not subsequently made during data collection for either the midline or the endline.

More general comments on decentralization and the health sector included:

• Weariness over the length of time that was being taken over bringing about decentralization of health systems: 'We continue to await the euphoria of decentralization' (District Health Management Team (DHMT) member).

• A number of respondents pointed out that more knowledge was needed on how budgetary and decision-making powers would/could be devolved specific to HIV and health and which government bodies at which level or levels would have the final say.

• Decentralization (to the extent it had happened by 2017) had ushered in more involvement of village and Ward level committees. Zonal, District Health Management Team (DHMT) and District Council respondents described how such community structures often sought to put forward their own health agendas, without any basis in data quality and verification.

• Decentralization might usher in more District ownership of data; however, unless digital and other health data are reliable, such ownership might be somewhat redundant in terms of usefulness in planning and improving service delivery and outcomes.

3.3 Key findings from the 2019 Kuunika midline evaluation specific to decentralization

• Information from midline KII was that there continued to be some reluctance on the part of some project consortium members to engage at District level. While some work was being done to change this viewpoint, much energy was being given to developing the project paper entitled Project Implementation Plan - operationalizing Kuunika sustainability (finished on 31st July 2019). This diverted time and capacity away from District engagement. This was considered to have been a (further) missed opportunity to engage properly with the District level.

• The unlocking of all four 'key capabilities' as set out in the 2019 - 2020 Kuunika sustainability causal pathways cannot genuinely and sustainably be achieved without attention to support at all levels: training, data management, data use and development of a knowledge management organizational culture.

• As at the baseline, the midline found very little use of financial or HRH data by those supported by the project, in part for sure reflecting the limited nature of fiscal decentralization to date.

• The development of the Local Authority MIS (LAMIS) was described as a potentially valuable tool for decentralized ownership of data, not least because information is that it will include a number of databases, not least DHIS2.

• A range of views were expressed on the relevance and usefulness of the DHIS2: o Respondents on DHMTs and at the District Councils referred to the District Implementation Plans as opportunities for using DHIS2 data to provide tailored and specific evidence to inform planning and budget allocations. o There continued to be a widespread perception of ‘data gathering for data extraction’.This was a consistent theme at District level. Both EMR and DHIS2 data were perceived to ‘funnel up’ data from facilities and districts to the national level. Such extraction and exclusion were felt to inhibit the use of analysed data for administrative, service delivery or planning purposes, and limit opportunities for the growth of evidence-based planning. o Less than one in five (18%) of all midline respondents used DHIS2 to look up data. A common concern was that the DHIS2 contains ‘too many indicators’. A majority of District respondents felt that the DHIS2 was ‘one size fits all’ with no scope for districts to tailor it to include indicators of particular relevance to their work or geographical area. o Poor quality of data was seen as having repercussions for district level planning, including the ability to argue robustly for budget and human resource allocations The quality of routine data was said to be a fairly regular topic of concern at district level meetings where DHMT, other clinicians and/or District Council staff worked together. There were concerns at national, zonal and district levels over the time lag between data collection and feeding into the DHIS2 system.

• Key to genuinely effective decentralization is engagement with ALL levels of the health system. Therefore, a midline finding of concern was that most Health Facility Advisory Committee members had inadequate knowledge of data collection and use and could not independently determine if data were of good quality. HSAs also described barriers to knowledge and use of data, despite being key partners in data collection for HIV (at the time, the Kuunika use case) and other data. HSAs were not included in any data use decisionmaking at Health Facility level. The majority of HSAs had not been trained on data entry.

One point discussed both at the midline and more coherently during endline data collection by key informants with a longitudinal perspective on the entirety of Kuunika, is that in 2019 a 'convening point', i.e. an organizational center and management hub, did not really exist. The development and strengthening of the MoH Digital Health Division is now seen as an integral component of digital data governance and national ownership. The same key informants were less positive about the extent to which such organizational framework building has so far been expanded down to District level.

As from November 2019 the core package work was extended into phase 2, with expressed increased attention to sustainability (including at District level). District Health Offices (DHOs) were then discussing how Kuunika could best support them to address District-level priorities. At the time it was planned that DHOs were to be more fully and actively involved in the project. Indeed this was a midline finding and a recommendation followed to ensure greater engagement of DHOs.

The 2019 Kuunika Implementation Plan notes that 'Feedback provided by subnational MoH during planning meetings for the final year work schedule and budgets pointed to the lack of full awareness and leadership involvement in implementation of activities. To address this, the Kuunika secretariat also embarked on a consultation progress with the five Kuunika districts to share the activities planned for 2019/2020 and receive feedback on district-level stakeholder priorities' (p. 3)

Of relevance here is that such consultation was to occur at least 2.5 years into project implementation; this begs the question - why not during planning or inception phase? The 2019 midline evaluation amplified such lack of dialogue and effective ownership and that is a feature finding of the endline too. Such incomplete focus on decentralized structures has deep roots, in that the June 2018 Kuunika core package pivot to 'unlock the 5 key capabilities' did not explicitly refer to District engagement, ownership, governance.

Nonetheless, one clear sign of progress at this stage of the project is that towards the end of 2018/early 2019, more District activities were underway, because recruitment of Kuunika officers to be sited at DHOs had been completed. The data use campaign was launched. So, District activities were achieving some momentum In addition, at the time of the midline there were indications of Districts becoming slightly more active partners in Kuunika, with DHO staff members feeling they had space to discuss addressing context-specific priorities.

Post-2019 opportunities specific to entrenching structures and systems for digital data at District level appear to have been missed - or more feasibly, been (partially or wholly) overtaken by the Covid-19 pivot.

4. Special Study 3 findings

4.1 General findings from the 2021 endline

Many 2021 findings echo, indeed amplify, those from the baseline and the midline in terms of lack of project involvement in the development of truly decentralized, less vertical systems of digital (and other) data collection and use. 7 Of course, such governance and often politicized issues are beyond the mandate of any one program. However, a significant number of 2021 key informants at all levels (national, District and Health Facility) see Kuunika as a program that has always primarily supported national digital data structures, while to an extent having previously been more visible and active in Districts. District engagement includes supporting EMR use and maintenance, providing considerable training, undertaking data use campaigns, providing facility dashboards in Zomba, and so on.

Key informant interviews: views on Kuunika's engagement with Districts - looking across its implementation 2017 - 2021

• Kuunika wasn't initially plugged into the Districts re. digital data use, ownership. The incomplete decentralization structures didn't help. Initially the Kuunika consortium was almost a parallel MoH, without any attention to decentralized health + data. (National level key informant)

• The original consortium didn't recognize that Kuunika is a project FOR the MoH + the GoM. Communication between District and national levels was lagging + lacking - Kuunika was somewhat designed in a vacuum, without thinking of communication channels between the project + Districts. The initial big focus in Kuunika was data - not which entity/individuals had access, ownership, etc. Later focus was more on patient outcomes + use of data to support service delivery; that necessitates proper District buy-in if improvements in service delivery are to be achieved. (National level key informant)

• When Kuunika was being designed, the GoM said it was decentralizing the health system, but most decisions + human resource management continued to be made at the central level. There is a centralized 'push system' for sure in health. Most projects at District level have to work very closely with the MoH, even if more focused on the Districts; this is true for Kuunika. (National level key informant)

• For sure Kuunika has provided significant training, hardware (e.g. initially EMR support; latterly e.g. facility dashboards), technical support, etc, but it has not sought to systematize such inputs, i.e. to help to build people's understanding of why it is so important to analyze and use data. (DHO staff member, Kuunika District)

• At Health Facility level, Kuunika is often seen as distant and disengaged, before and during the Covid-19 response. (DHO staff member, Kuunika District)

One key Special Study 3 finding is that the cumulative experience of the Kuunika evaluation shows that there needs to be greater genuine ownership of data at the District level. This necessitates buy-in from all partners working in any one District and nationally, challenging in a donor landscape currently as fragmented as Malawi; this has obvious implications for effective coordination of digital health initiatives. The District Health Offices should ideally be integral partners in data collection and use.

In addition:

Development of a knowledge culture, where individuals, institutions and organizations genuinely acknowledge and internalize the understanding that data collection and use will improve health planning and service delivery, and where stakeholders work together effectively, is hampered by what is perceived to be the lack of performance/results-based management in the Malawian public sector (including health). Kuunika has not been able to make much headway in supporting acknowledgement of the need for, or the institutionalization of, such a knowledge culture.

7 Review of the 2019 Vital Wave report entitled Assessment of EMR Systems in Malawi. Prepared for the MoH, Republic of Malawi. Initial Landscape Assessment indicates many similarities in its areas of focus and those of this special study (e.g. lack of knowledge culture; lack of support to Data Clerks; lack of attention to demand-side/ equity issues). One interesting difference is the absence in the Vital Wave report of any detailed consideration of District-level data structures and systems.

Information is that there are currently (from mid-2021 onwards) changes being made at national government level specific to what might be seen as a (re) centralization of digital health data management and (re) focus on DHIS2. For instance, Digital Health has been moved back into the MoH Department of Planning and Policy Development.

There is opinion that this might dilute a developing center of excellence in terms of trained personnel. The move might limit, even reduce, Districts' and health workers' capacity to review and make speedy use of data; such moves might also have impacts on the building of digital data systems and knowledge culture at District and facility levels

The project's community engagement is also considered to have reduced over time, apart from those few key informants at DHOs who discussed OHSP and IDSR work during the pandemic and how the community component has been important for gathering key surveillance data.

So can we define the constraining and supporting factors shaping Kuunika's engagement at District level and below?

Factor analysis

• Limited space for decentralization of the Malawian health system

• Limited financial decentralization + budgetary control

• Limited attention to District health structures in Kuunika Proposal and initial design, despite (?) there being Malawian consortium partners

Constraining Factors

• Kuunika has provided very considerable training on DHIS2 across the project Districts, but limited integration of data actually collected into District planning

• DHIS2, supposed to be primarily a District-level digital data system, has limited user base at DHOs + in Health Facilities, and also at community level

• The view prevails that data collected at District + lower levels, by whichever partner, are primarily extracted in a vertical process, with insufficient discussion or joint planning

• Evidence of slowly expanding appetite for development of a knowledge culture in District health systems through use of digital data + DHIS2, supported by the project

Supporting Factors

Looking forward

o To an extent supported since 2017 by Kuunika and other projects' inputs, e.g. training of Data Clerks, appointment of Kuunika HMIS Officers at DHOs, EMRs, dashboards, WhatsApp, Cluster meetings, DIP engagement o Data 'super users' are pathfinders + their engagement has been prompted/ expanded through Kuunika o Engagement with HSAs, CBOs, communities + to a lesser extent health Committees primarily positive but relatively limited in terms of outputs

• Engagement with District Councils, under whose remit come the District Health Offices + DHMT, + support through Kuunika-funded staff membersthose staff have facilitated access to digital data for a wide range of District officers, whose direct involvement with the data remains limited; Council officers especially often view DHO DHIS2 data as 'challenging'

One big lesson is that a project cannot just design digital data systems (or expand those) without properly thinking of who (up and down) will use them, how they will be enabled to use them and what the capacity development needs will be.

If it were possible for Kuunika phase 2 to make an input into the development of a knowledge culture at District level, this would surely enhance opportunities for greater and more effective use of digital data in planning and service delivery.

The Blantyre Prevention Strategy (funded by BMGF and on which Cooper/Smith is a partner) is said to have been designed based on lessons learned about Kuunika gaps. This new program has the DHO as the lead, from the start. 'We rethought the process, based on Kuunika challenges regarding decentralization'.

Therefore, while there must be attention to limiting the further proliferation of digital data pilot projects, the BPS may have lessons that can be scaled up to other Districts.

4.2 Kuunika, the Covid-19 pandemic and the Districts

Once the pandemic hit Malawi, project attention and energy focused on developing the C-19 digital data response. Thus in March 2020 there was a further Kuunika pivot, answering the Ministry of Health’s request for assistance with national data collection and monitoring of the Covid-19 pandemic. This appears to have shifted Kuunika’s focus back to the center As yet there is no documentation on whether and how such work might inform future Kuunika District-level engagement and support.

One quite consistent view from both internal project and external national key informants is that Kuunika was catalytic in supporting the Ministry of Health to deploy and pivot to collect Covid-19 data. This has all been done using open source systems and is seen as an example of the importance of the interoperability layer.

Information from Cooper/Smith is that: Covid shifted priorities and the timeline of activities, but the re-evaluation of remaining activities did not flag a permanent change. This may be supported by the fact that our role in some of these activities has shifted from development to technical guidance and oversight.

In addition and specific to whether Covid-19 response inputs from Kuunika have relevant longer term lessons: there have been a lot of lessons learned in coordinating the scale up of a national tool that supports different segments of a system e.g. surveillance, clinical and laboratory services. [There are also] lessons on user support, partner engagement, data quality, data access, and governance.

This graphic is copied from the June 2020 Kuunika document Sustainability Phase & Covid-19 Pivot: overview and sets out key areas of support.

Figure 3: document Sustainability Phase & Covid-19 Pivot

The updating and rollout of the One Health Surveillance Platform (the OHSP, repurposed from the existing Integrated Disease Surveillance and Response (IDSR) system) represents a major component of inputs provided to and working with Districts (including Health Facilities and communities)

Experiences of its implementation are varied among DHO and Health Facility key informants. Relevant here is that whatever individuals' perspectives on the national Covid-19 response and the actions taken to collect data, all those with any project-specific information felt that Kuunika had sought to make speedy, flexible and responsive inputs in a very challenging environment and was to be commended for such support

Key informant interviews: the OHSP, Covid-19 data collection + use

• In April 2020, Kuunika deployed some 30 phones to IDSR co-ordinators + oriented them on capturing Covid data. There was orientation of Data Clerks on how best to capture the data. Some of the data sets on the OHSP allowed us to gather information on cases, which were entered on the system e.g. contact registration + follow up - Kuunika helped us with tracing of C-19 through contact registration and follow-up, working with the HSAs; OHSP captured data, working with IDSR Focal Persons in the communities. Kuunika also helped us with clinical case management, because data on hospital admissions could be captured from time to time. I think Kuunika made some inputs to orienting data people at each facility? (DHO staff member, Kuunika District)

• Kuunika came to see us collecting data for Covid-19, from the Health Facilities. I haven't yet had any feedback on the data collected or what they mean for us in the pandemic. There is one Data Clerk here helping in the data collection for Kuunika (DHO staff member, non-Kuunika District)

• Data collection has been a very major challenge. I'm not sure what Kuunika has done to help with this (or what they've tried to do, because the situation continues to be bad). Service delivery on all fronts (i.e. not just for Covid) has been delayed, so data collection has been compromised. Generation of data and reports have been delayed, incomplete, weak. Vaccination data for C-19: so much information has not yet been uploaded. Data should be electronically entered, but this was delayed, there were the usual network problems, etc, so we resorted to manual data entry. There is a huge backlog. (DHO staff member, Kuunika District)

• We have had to use the existing structures, the IDSR Unit and its staff. There was no co-ordination, so efficacy was limited. Each partner comes with its own ideas and budget; there should be co-ordination at the MoH. (DHO staff member, Kuunika District)

• There has been quite some activity. There is a person called the District Data something under Kuunika [this is the Data Use Facilitator] and there has been a lot of training of HMIS Clerks and IDSR staff to collect Covid-19 data for Kuunika. (DHO staff member, Kuunika District)

• We haven't had any training here on Covid-19 data entry or use. But the DHO has given us some information, through WhatsApp. (Health Facility staff member, non-Kuunika District)

• We have had no extra support to collect Covid-19 data, from Kuunika or any other project. It would have been good to have that support, e.g. on how to collect Covid-19 data, plus specific indicators and a checklist. (Health Facility staff member, Kuunika District)

• There hasn't really been any support from the District or any project during Covid in terms of data collection, other than the DHO giving us order and inventory forms. There's been no support on data analysis and use and review meetings have been cancelled (Health Facility staff member, Kuunika District)

Review of 2020 and 2021 Kuunika District Quarterly Support Reports, training reports and District Activity Plans reveals the following:

• Very considerable inputs have been made by the project in terms of supporting data collection in Districts, through training of IDSR staff, Data Clerks and others

• The post of Data Use Facilitator has been created and one such person has been working in each of the Kuunika-supported Districts

• Detailed Covid-19 data have been collected

• Despite such support, considerable data backlogs have occurred, due to volume of data to be entered on to OHSP (which some IDSR Focal Persons find hard to use) and lack of time for many Data Clerks. This means that major exercises in data cleaning and back data entry have had to take place, e.g. in Lilongwe

• Such backlogs will of course have implications for completeness and overall quality of Covid-19 data

The Kuunika District Support Quarterly Report (for Q2 2021) states: Blantyre district accumulated a huge backlog of 31,094 COVID-19 vaccination data [as of 10th April] due to increased workload and inadequate devices to capture data. This affected the overall quality of e-vaccination data.

• Despite the considerable inputs documented in Kuunika reports, accurate knowledge of its work specific to Covid-19 response does not appear to be widespread at District or Health Facility level (and information at second hand is that the same is true in many instances at community level)

• For some District and Health Facility key informants, the Covid-19 digital data intervention appears as another vertical, data extraction exercise, whose main focus is not to support optimal and equitable service delivery or to facilitate increased access to and use of evidence.

4.3 One very practical example: has Kuunika been involved with the development of District (health) Implementation Plans?

This is a topic that receives a range of significantly differing responses. We have been prompted to consider it here because of key informants' sharply differing views of Kuunika inputs to the DIP development process. As elsewhere, the necessary caveat is that no single project can be expected to achieve all necessary change.

The evidence indicates that the project has engaged with DIP development in Zomba District; elsewhere key informants are less certain as to any involvement

One striking finding is that there is minimal discussion of any aspect of DIP in national health or digital data for health strategies. This absence does not provide an ideal foundation for systematizing evidence-based planning at any level, but of course particularly in Districts, Health Facilities and communities

Literature review

• New annual (2022) District (health) Implementation Plans (DIP) are currently being developed

• Clinton Health Access Initiative (CHAI) is a major partner in support of DIP development, as is UNICEF

• Few national strategies even refer to DIP development or implementation processes; therefore, presumably none views these as core components of effective planning + service delivery

Key informant interviews

• 'Before 2020, DIPs were based on each program co-ordinator's degree of commitment. Districts now have a very comprehensive spreadsheet, they do bottleneck analysis. The DIP get budgetedsaying what is GoM/ partner/not funded. However, one major challenge is that the Treasury does not disburse what is requested, plus it is always a struggle to harmonize funding coming through partners, i.e. to get them to co-ordinate. We are seeking to get those District partners involved in DIP development. ' (National level key informant) o The sole reference in MEHIS 2017 - 2022 is discussion of work towards the achievement of the HIS Strategy 2011 - 2016 Objective 8, where HIS data are stated as having been used during DIP development o NDHS: no reference o The undated (but 2016 or later) MoH Data and Digital Priorities: Digital Health for Universal Health Coverage is similarly silent

• No information has been forthcoming as to whether the HSSP III, currently in development, will address any aspect of DIP

• Neither the Blantyre Prevention Strategy Condensed Strategy document nor the BPS Fact Sheet refers to the DIP as a planning tool using digital (DHIS2 + other) data, nor addresses the extent to which either the 2021 or the 2022 DIP does or does not make use of DHIS2 data + Bottleneck Analysis to inform service delivery planning + implementation

• 'Kuunika has helped, directly, to develop the new DIP, helping to capture important aspects of data, in terms of team engagement too.' (DHO staff member, Kuunika District)

• 'The DIP is renewed on an annual basis. DHIS2 has become increasingly useful, in terms of developing targets + indicators; each co-ordinator is expected to draft indicators for his/her area. There isn't yet comprehensive use of DHIS2 data (annualised) to address gaps, etc, for the next annual plan. We use DHIS2 as the databank. DIP are not publicly available; they're working documents. We don't have enough funds to disseminate the DIP to the communities, which is a shame + means Committees + CBOs don't get to be part of the debate. Kuunika involvement? Not sure if there is any - certainly not here, but in other Districts?' (DHO staff member, nonKuunika District)

Key question 1: do any DIP show evidence of DHIS2 data supporting evidence-based planning and/or better evidence use? Are Bottleneck Analysis (BNA) findings actually applied to planning for improvements in health service delivery?

One national key informant made the following points, not specific to Kuunika, but revealing about the current status of quality data availability.

'At Health Facility level data quality is a major, major issue. We have sought to use evidencebased planning and bottleneck analysis looking at both the demand and supply sides. The UNICEF Tanahashi Framework is used by one program to analyse the bottlenecks - but in Malawi the data are so lacking or so weak, the categories frequently don't make sense. Quite often the data are reported at above 100% or below 0%. The DIP process reveals that so many data sets just don't add up, in any way. In almost every District the data are so poor, it's more lip service on evidence-based planning.'

The last point is true also for Districts supported by Kuunika, according to the key informant.

Another national level key informant, also with experience of District-level data capacity, stated:

'Some data don't make sense when bottleneck analysis is done - if data are above 100%, then quality is so very poor. The DHIS2 has so many gaps, for indicators and also across time periods - so completeness of data is lacking as well as quality.'

Key question 2: do Kuunika-supported Districts provide any indications of greater engagement with evidence-based planning?

There are some indications of this. Thus the Q2 2021 Kuunika District Support Quarterly Report indicates that the Data Use Facilitator for Zomba (a post funded by Kuunika) participated in a threeday DIP consolidation meeting...supported by CHAI. The main activity was to consolidate all the programme activities into the main DIP template and fill in the performance of different indicators.

Factor Analysis of the 2019-2020, 2020-2021 and 2021-2022 Zomba DIP (District Planning and Budgeting Tool) in terms of use of DHIS2 data + Bottleneck Analysis (BNA)

Constraining Factors

• 2019-2020 DIP does not include BNA

• Lack of clarity as to why the DHIS2 indicators included in the Excel spreadsheet on BNA (PS2) for 2020-2021 + 2021-2022 DIP have been chosen

• Many District programs + their indicators are not (yet) on DHIS2

• It is unclear whether there is any automated link between DHIS2 + PS2 + the BNA

• 2021-2022 PS2 spreadsheet shows that only circa 50% of the DHIS2 indicators listed have a national target

• Many indicators in the 2020-2021 and 2021-202 DIP state 'no activity logged against this indicator'

• Most of the DHIS2 indicators on PS2 are not in the BNA Application, making for limited analysis + read across

• The 2021-2022 DIP lists more indicators performing 'equal or below' the national target measures than above

• It is not possible to gauge the quality of the data included in the DIP

• Issues of disaggregated data or an equity focus appear not to be part of DIP planning (or implementation?) - are these essential levels of complexity too much to ask for at present?

• It is unclear whether the BNA has informed inclusion of new DIP activities

• The extent to which the DIP has been implemented year on year is unclear, yet tracking that is key

• Compared to the 2019-2020 DIP, the subsequent 2 years' Plans indicate more engagement with data + more contextualizing of data to address Zomba priorities

• The BNA uses DHIS2 data + informs identification of root causes

Supporting Factors

• The fact that the DIP is now populated using charts based on DHIS2 + other data is positiveagain, no facility to gauge quality of the data used

4.4 Kuunika evaluation hypotheses relevant to decentralization: findings from Special Study 3

Here we track the entirety of the evaluation work from 2017 with regard to the hypotheses that underpin the Theory of Change and the originally five, now seven, evaluation questions. Five of the 21 hypotheses have particular relevance for this Special Study.

The degree of organisational and political decentralization can affect use of evidence in decision making (hypothesis 5)

2017 (pre-baseline planning + baseline): weak and conflicting KAP shows some culture in data use, uptake and challenge (probably reflecting large % of clinicians in the sample), also increasing at higher levels. The qualitative research shows that zonal and district decision makers (both generalists and health sector) are unsure of the overall parameters of decentralization e.g. will the zonal health teams continue to exist? Will the generalists without health expertise and also the many 'new' levels, e.g. village and Ward committees, have more sway in deciding annual foci for health?

2019 (midline): It has been difficult to gauge the extent to which the ongoing decentralisation (a process that has been ongoing for twenty years or more) is influencing the use of evidence in decision-making. The few District level respondents available to the midline did mention increased use of DHIS2 to develop District Implementation Plans; its expansion was considered potentially positive for creation of increasingly detailed District plans. The development of the Local Authority MIS system was described as a potentially valuable tool for decentralised ownership of data. As at the baseline, the midline found very little use of financial or Human Resources for Health data, reflecting the limited nature of fiscal decentralisation to date

2021 endline and Special Study 3: The hypothesis is assumed to be true, but has never been tested. The Malawi health sector is de jure quite decentralized, de facto not so much. Still to find evidence of greater use of data in DIPs, etc. How to break this circle? Covid-19 monitoring experience has possibly reinforced the sense of a centralized, push system - and perhaps led to greater acceptance of this by some at District level? The pandemic does not appear to have led to more democratization/localization of data capture and use Districts continue to be out of the loop for use of evidence for decision making - and to an extent in its actual use.

Policy making is often messy and opportunistic - ‘a disorderly set of interconnections and back and forthness' (links primarily into hypotheses 7 and 17)

Hypothesis 7

2017 (pre-baseline + baseline): There is more than 1 route by which better data influence policy and practice – by feeding into higher level (especially, for HIV, international) policy making and by informing local planning and decision-making.

2019 (ML): the formal/international policy making level dominates in HIV in Malawi. Need to dig more into ‘informal’ and messy decision making in the districts in the special studies? Pretty messy for sure, given decentralisation (or more precisely, its absence, despite 'formal' enactment) and also external partners' influence.

2021 endline and Special Study 3: governance issues are relevant here. Data collection indicated that analysis and modelling of Covid-19 data by Kuunika showcased the power of data amongst central planners and excited a lot of debate.

Hypothesis 17

2017 (pre-baseline + baseline): by limiting access to the improved DHIS2 data, the role of other groups (outside the formal health system) that help accelerate change may be restricted e.g. the District Policy Planning Officers, who don't know if they'll have such access, but seem to be increasingly important pivot points in the decentralised structure.

2019 (midline): the key pivot point here in terms of developing a knowledge culture that supports policy/ practice effectiveness is for individuals and institutions to realise that effective data use and evidence-based policy making are likely to result in enhanced service delivery, better health outcomes and ultimately support clinical health workers, health planners and data entry personnel in their work. In other words, how to change 'institutional norms' and make engagement with data use a truly positive attitudinal and behavioural action.

2021 endline and Special Study 3: has such 'back and forthness' continued under the weight of pandemic response? Has the pandemic speeded up any policy/governance actions and/or shifts? Are there signs of greater/lesser order and coherence in policy development processes, not least given the urgency of dealing with C-19 in 2020?

Hierarchical management of information, organisational silos can limit access to data and its use. Divisions of responsibilities and ‘silos’ can also limit consideration of evidence (hypothesis 15)

2017 (pre-baseline + baseline): widely/more accessible (within the health system) data on DHIS2 may break down these hierarchies

Depends though on who has user rights and relatively easy access to the system. The 2017 baseline found very little use of DHIS2 at facilities and extensive views that data collection is extractive. There were few obvious feedback loops. There were clear ‘job silos’ - Data Clerks enter data but do not analyse; in-charges (supposedly) analyse but do not enter.

2019 (midline): while the health system is undoubtedly hierarchical, not least in terms of positioning data collection at the lowest level of a steep pyramid, that structure does not result in lack of awareness of the important work done by Data Clerks, or any demonstrated superiority from those with clinical and/or management training and experience. The Kuunika-led cluster meetings and QI sessions which we found at the midline to be well accepted could be interpreted as working by breaking down hierarchies and job silos around data use. By limiting access to the improved DHIS2 data, the role of other groups (outside the formal health system) that help accelerate change may be restricted.

2021 endline and Special Study 3: it appears that Kuunika has not really expanded focus specific to (more? less?) hierarchical management of data. There are indications that Covid-19 response/data collection and use has become a silo and for some District and lower level key informants has represented challenging demands.

Individuals are empowered through access to data (hypothesis 18)

2017 (pre-baseline + baseline): DHIS2 will be more effective if it is designed with the users’ needs in mind

2019 (midline): limited evidence of such shifts (empowerment is interpreted in this context as meaning personal job satisfaction, acknowledgement that access to data improves the quality of one's work and opportunities for advancement). No evidence has been found of individuals (as compared to facilities) being awarded DHA Certificates, receiving commendations or other professional satisfaction because they have increased access to data.

There was expressed concern about achieving an appropriate balance between greater access to routine health data and what might be required of individual health workers. Empowerment might come with attached demands for increased data productivity, more analysis, etc., so whose empowerment would it actually be?

'Super data users' very clearly understand and celebrate the opportunities given by greater collection and analysis of routine health data and are empowered personally (this includes the one Data Clerk who is such a user, who has led data quality improvement activities at Health Facility level, with expressed support from all clinical colleagues). So too do policy makers at national level, whose position in the health hierarchy is already empowered.

As at baseline: DHIS2 will be more effective if designed with users' needs in mind. The midline found that Kuunika steps to develop DHIS2 using user-driven design approaches appears to be working to promote interest and anticipation as to what can be done + in connecting with a few self-propelled 'super (data) users'.

2021 endline and Special Study 3: what might have changed as a result of the pandemic - any greater horizontal empowerment? The pandemic has resulted in (temporary?) cessation of WhatsApp groups and review meetings. Access to DHIS2 does not seem to have greatly expanded. How might the slowly expanding group of 'super users' become more integrated into District digital data systems (without placing unreasonable demands on them)?

5. How has Kuunika performed against aid effectiveness principles?

In this section of the report we apply a broader perspective, to consider Kuunika's inputs throughout the life of the project and the extent to which its inputs, intermediate outcomes and outcomes may have contributed to aid effectiveness. The 2005 Paris Declaration foregrounded the central importance of aid effectiveness principles being agreed, adopted and applied.

Reflections on Kuunika’s experience in the Districts against the benchmark of aid effectiveness principles

Aid effectiveness principle

1.

OWNERSHIP:

Developing countries should be owners of their development

Assessment of Kuunika’s implementation experience

● Kuunika has throughout worked with the central MoH, which is the owner of the project. However, where should national ownership be sited? Only at the central level, or also at District and lower levels, where the great majority of health service delivery takes place?

● There is considerably less ownership in the Districts, where throughout the lifetime of the project issues have been raised of data extraction, multiple (and sometimes incompatible) reporting systems, vertical programs reporting to the national level, donor agendas sometimes outweighing District priorities in terms of data collection and use.

● Kuunika's position at central level has meant that its inputs have been aligned to MoH and by extension, GoM, policies and systems.

● Kuunika's support to the development of the 2020-2025 National Digital Health Strategy reflects project commitment to alignment (but see e.g. section 2.4 for discussion of this Strategy)

● Major project technical, financial and human resource support to the development of the MoH Digital Health Division is another major benchmark of adherence to the principle of national alignment

2. ALIGNMENT: Development assistance should be aligned to country policies, institutions and local systems

● So too is the development of the Covid-19 digital data response, where Kuunika, other development partners and the GoM (primarily but not solely the MoH through the Digital Health Division) have worked together

● But such achievements are less supportive of true alignment between central and lower levels of the health system, digital data collection and use at District, Health Facility and community levels.

● This begs the question: if Malawi has a highly centralized health system, including for digital data, what is the most effective role of any one intervention such as Kuunika in seeking to expand the definition of alignment, when there is evidence that in 2021 Districts are not yet full partners in the digital data ecosystem?

● Furthermore: aid effectiveness in the context of alignment specific to digital data systems should surely include focus on optimal alignment of sub-national and national structures - Kuunika has not yet optimally supported such work?

3. HARMONISATION: Developing countries and partners should

● Evidence indicates that a '[Health] Sector-wide Approach (SWAp) Revival' Concept Note is in development, an activity supported by the GoM + a number of partners; this must signal a degree of harmonization intent to