Executive summary

Gavi has contracted Mott MacDonald to undertake a mapping and assessment of learning and performance management approaches of frontline health workers. The purpose is to inform Gavi’s support to countries during its new 5.0 strategy, to build health workforce capacity to address the problem of high unreached (zero dose) children and under-vaccinated populations, as well as preparing health systems for COVAX and other vaccine introduction

Since the early 2000s, there have been large increases in donor financing of human resources for health (HRH) - currently estimated at estimated at $4-6 bn per annum1 - yet much investment is not coordinated with national health workforce plans and there has been little analysis of impact. COVID-19 has brought about not only new imperatives but also opportunities to make use of new technologies to disrupt the way things are done. The need to limit face-toface interaction, upskill large numbers quickly and cope with even more pronounced economic pressures worldwide speak to the need for more efficient and impactful approaches.

Focusing on performance and quality of care

The Lancet Commission on High Quality Health Systems asserts that moving to a high-quality health system is primarily a political, not technical, decision. Governments should start, it suggests, by establishing a national quality guarantee for health services, specifying the level of competence and user experience that people can expect, and against which governments would be held to account. We know from evidence reviews that knowledge is necessary but not sufficient for provider performance which is influenced by a range of factors including sufficient budgets, supervisory systems, staff motivation and retention, supply chains and back-up health facility readiness. Provider performance, therefore, cannot be considered in isolation from the wider context of health system reform, the need for better governance and accountability, and a more people-centred approach.

Content-driven, one-size fits all approaches are effective only for specific situations and need to be complemented by more learner-centred, layered approaches in others. Applying the evidence at scale becomes an implementation challenge, which is why health sector coordination, and combinations of approaches, is so important. Even the best, most innovative solutions will not be effective without appropriate workforce strategies, health financing, governance and health data in place.

Review of the most promising examples of scalable innovation suggests factors in success relate to the extent solutions are integrated into existing systems, institutional, behavioural and political contexts and resource environments. This suggests that broader, multifaceted, health systems approaches to performance (ultimately impacting quality of care and health outcomes) are needed.

Evidence from systematic reviews

A recent systematic review of provider performance strategies confirmed that multifaceted strategies targeting infrastructure, supervision, other management techniques, training and group problem solving tended to have large effects. Combining training and supervision had larger effects, for example, than use of either strategy alone. Group problem solving including “collaborative quality improvement” strategies were shown to be particularly effective.2 As for the training attributes most associated with effectiveness, educational outreach visits at health

1 Personal communication with Jim Campbell, WHO, 13 January 2020

2 Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review. Alexander K Rowe, Samantha Y Rowe, David H Peters, Kathleen A Holloway*, John Chalker, Dennis Ross-Degnan. Lancet Glob Health 2018; 6: e1163–75. Published Online October 8, 2018 workers’ sites were found to be more effective than in-service training, which was, on average, more effective than peer-to-peer training and self-study. In-service training that incorporated clinical practice tends to be more effective than training without it, and training that at least partly occurred at health workers’ routine worksite (on-site training) tended to be more effective than training that was entirely off-site. These findings likely relate to on-site training being more likely to be immediately relevant to a provider’s role with a greater focus on hands-on practical skill building (see Section 2.3). These are aspects that any digitalisation of training should try to replicate where possible.

Integral to any approach must also be effective monitoring of health worker performance and quality of care. This is especially true because of the highly variable effectiveness of strategies from one context to another, and the need to detect specific clinical/inter-personal aspects of service delivery that need improvement. Monitoring data can also be shared publicly to help accountability efforts needed to ensure high quality systems.

Best practice from implementation experience

Some important principles of best practice for interventions to improve health worker performance, many somewhat overlapping, are as follows:

➢ Using a blended learning approach, where components can be layered and tailored to requirements, generally works best; finding the right balance between face to face and virtual approaches might need trial and error to get right;

➢ Using different media and formats tailored to context increases success of training

➢ Co-designing with health care providers themselves allows training to focus on practical issues rather than theory, thus becoming more relevant and immediately transferable to health providers’ practice

➢ Including problem-solving approaches and collaborative quality improvement addresses specific identified challenges rather than relying on approaches which deliver standardised content

➢ Group dynamics, peer support, buddies, and communities of practice are an important part of skill-building and norm setting

➢ Ensuring co-production with government is more likely to lead to ownership and government buy-in and sustainability

➢ Linking training with accreditation as part of a lifelong learning approach increases uptake /completion rates and ensures better alignment with national health workforce strategies.

These principles are expanded, and illustrated with examples, in the main report.

Use of digital approaches

The above-mentioned systematic review of performance found that the average effectiveness of improvement strategies that included mHealth/ICT (information and communication technology) were not particularly high, with a median improvement of just 1 %-point for ICT alone, and 8 %points for ICT in combination with other intervention components. Some digital education has been conceptually pedagogically weak, with more attention to the technology than learning impact. However it is a fast-evolving field and WHO has recently published a discussion paper “Digital education for building health workforce capacity” with evidence pointing to digital education being at least as effective as traditional education in improving health professionals’ knowledge, skills, satisfaction with the educational approach and professional attitudes Certainly more use of blended approaches that include digital offers potential to reduce disruption to service delivery from provider absence, reduce the massive amounts possibly wasted through conventional classroom training and harness the power of digital and data analytics for more tailored learning experiences.

The following lessons have emerged on the relevance, feasibility and acceptability of technologies commonly used in digital training or data exchange.

➢ Functional requirements include ability to work in a low bandwidth or offline environment, ease of use and low costs

➢ Participants generally prefer to access content on their own devices rather than carry separate devices (although in some settings health providers do not own their own devices)

➢ Mobile phone training through Interactive Voice Response (IVR) can be an effective complement to classroom training for CHWs, even in fragile states

➢ Multi-purpose systems which allow functional integration are far more likely to be used

➢ Support for change management and technological troubleshooting, both for MoH officials and for health workers themselves, is vital to allow transfer of ownership

➢ Building modular content using existing “global good” platforms supports integration, interoperability and sustainability

➢ Open source is desirable but does not mean solutions become “free”

Better, Fewer Metrics

The issue of measurement of performance strategies’ impact is widely seen as an area in need of attention. It is essential for personalised learning, course correction, ensuring strategies can be optimally tailored to context and for ensuring cost-effectiveness of investments. While tailored, personalised learning is seen as priority, the challenge is delivering this at scale. More than ever, pedagogical experts and software developers need to work closely together, to ensure that the power of digital and data analytics can be used to accurately pick up individual weaknesses or difficulties and support tailored learning. This includes greater sensitivity of measurement with feedback loops and remedial action to ensure learning needs are met.

Per diems have become a de facto salary top up in many settings yet simply incentivise attendance at training rather than impacting performance. In the context of low resource settings and constraints on budgets to adequately pay the health workforce, there is certainly an argument for official development assistance (ODA) funds being channelled to frontline health workers if this can leverage quality. Digital technology can be used for virtual, direct observation, as part of more tailored learner-centric learning approaches, as well as for community feedback. Budgets currently spent on training could be partially redeployed to incentivise better quality care, based on the “better, fewer metrics” and people-centred measurement advocated by the Lancet Commission.

Sustainability

All the case study countries are embracing digital health in some form, albeit in a more limited way in the most fragile, Somalia and DRC. There are some novel and excellent approaches, though even the most established are still struggling to find a truly sustainable funding model. Digital learning and performance management need to be seen within the wider lens of digital innovation which faces the same challenges as any new intervention in global health.

Sustainability is as always, the biggest. Ultimately until a major intervention is embedded in a country’s relevant sector plan with funding provided by the government or by individuals it will not be sustainable. In countries such as Ethiopia and Rwanda, where the government takes a strong lead in setting the agenda, providing adequate funding for maintenance etc. and coordinating partners accordingly, country owned sustainable solutions are more possible.

Donors and implementing partners have an important role to play in helping ensure solutions align as far as possible to a coherent national plan - particularly in settings where capacity and digital maturity are less advanced. This means ensuring an aligned approach, investing in interoperable systems and ensuring that new innovations are compatible with the existing digital infrastructure.

Way Forward

In taking forward support for strategies which accelerate health provider competency development and performance, Gavi should consider its unique role in supporting solutions to achieve scale and gain traction at country level and globally. The following actions are suggested:

1. Join forces with other multilateral organisations such as WHO, UNICEF, the World Bank and The Global Fund using their platforms to support improved global and in-country coordination of frontline health worker interventions. This could include: o Advocating for more internationally recognised, nationally accredited, usercentric learning experiences including via the WHO Academy in collaboration with local ministries, professional councils and associations; o Investing in courses for vaccination managers and further language translations of successful courses; o Supporting ministries of health in fragile settings establish databases for health provider training to avoid duplication; o Running coordinated live simulation exercises with in-country partners e.g. for pandemic preparedness / vaccine roll out

2. In relation to digital learning and performance solutions, build on what exists and invest in further language translations, local adaptation and contextualisation, supporting interoperability and sustainable funding mechanisms for existing proven technological solutions; in particular, o Using blended, multi-faceted approaches rather than a single approach – opting for integrated digital solutions which address competency building, performance feedback, and accountability measures such as digital vaccination tracking; o Supporting initiatives such as Digital Classroom which can rapidly update, translate and roll out materials o Encouraging more partnerships between platforms such as CommCare and Community Health Toolkit in support of interoperability.

3. Ensure rollout of solutions for provider education / performance is country driven and aligned with wider support (through HSS grants) to address health workforce and other health systems constraints. This might include: o Providing technical assistance to countries to address staff turnover by developing career ladders for frontline health workers, with performance reward and recognition strategies, and investment cases for approaches to reduce wasteful turnover; Ensuring in-service training strategies have MoH buy-in and there is a clear link to career development; investment in a network of local partners for sustainability; o Fostering greater links with pre-service training institutions, and providing technical support to digitise elements of pre-service curricula; o Consolidating and leveraging more value from existing platforms, for greater economies of scale and to build familiarity with common platforms

4. Ensure that all innovative/digital investments are aligned to the country digital health plan if it exists, digital development principles and country cMYPs Consider Gavi becoming a signatory to the digital development principles. Where a digital health plan does not exist, encourage / support the development of one. In terms of aid effectiveness and coordination, ensure investments are aligned with and communicated to other stakeholders/donors; encourage co-financing from country government for implementation of digital technologies, and build up capacities of local Digitech companies through north-south business partnerships;

5. Channel support to district level capacity building for quality improvement initiatives This could include: o Identifying lagging behind districts and agreeing collaborative quality improvement initiatives - identification of the problem, root cause analysis, action planning, agreement on focused indicators, data collection and rapid feedback cycles (“Roadmap-Lite” approach); o Deploying a cadre of technically savvy graduates to be embedded within district teams to support digital elements within above initiatives, particularly troubleshooting use of digital data systems and producing data visualisations; o Capacity building of governments for in-house data analytics and sustainable use of data; o Supporting district-level management training, to build exposure to less hierarchical, problem-solving / coaching approaches.

6. Seek to measure impact in relation to the original problem that is being addressed using better, fewer, metrics that reflect quality of service delivery. This might include: o Ensuring regular monitoring of a limited number of indicators, rapid feedback cycles and two-way data flow that allows adaptation, layering and continual improvement of interventions; o Using mobile phone surveys for collecting rapid feedback from health service users on quality of care, missed opportunities for integrated care etc

7. Ensure better alignment of funding and incentives and consider innovative payment for performance mechanisms to replace training per diems, based on empirical evidence of service improvement.

8. Allow disruptive bottom up strategies that challenge traditional power dynamics, but explore the feasibility of pushing similar models down to lower tiers where nonEnglish/French speaking participants can engage in their own language. This might include: o Exploring learning laboratory approaches to distil and continually refine such approaches; o Considering a financial mechanism to support “bottom-up” action plans developed through this route via HSS grants, for example micro grants which alumni of such networks could apply for (with involvement of district health teams) for small scale collaborative quality improvement initiatives