2 Evidence on Healthcare Provider Performance

2.1 Healthcare Provider Performance

Healthcare provider performance has been described as a relatively broad construct that encompasses availability, clinical competence, responsiveness (providing patient-centred care), and productivity (or efficiency).5 We know from the various evidence reviews that knowledge is necessary but not sufficient for provider performance and that this is shaped by many more complex factors. A WHO global survey of LMICs on efforts to improve health worker performance found that health worker practices are positively related to health outcomes. However, health worker performance is influenced by a range of factors including sufficient budgets for training, supervisory systems, staff motivation and retention, supply chains and back-up health facility readiness.

The Lancet Commission on High Quality Health Systems, a major influential body of work premised on the notion that high-quality, people-centred care should be the raison d’être of the health system, asserts that moving to a high-quality health system is primarily a political, not technical, decision. Governments should start, it suggests, by establishing a national quality guarantee for health services, specifying the level of competence and user experience that people can expect, and against which governments would be held to account. It argues that incremental tweaks are insufficient and that a “major reboot” of health systems is needed.

One year on, the Lancet Commission reported that policymakers had agreed that it was time to look beyond marginal improvements and consider major reforms. Universal Health Coverage (UHC) could serve as an entry point to reimagine health systems that are fit for purpose, with strong financing but also new models of governance, provider training, service delivery, and community involvement. The Commission proposes that health systems be judged primarily on impacts, including better health and its equitable distribution; on the confidence of people in their health system; and on processes of care, consisting of competent care and positive user experience.

Provider performance, therefore, cannot be considered in isolation from the wider context of health system reform, the need for better governance and accountability, and a more peoplecentred approach. A more health systems focused approach will also counter the proliferation of many micro-measures which can indeed be deleterious to wider improvements. While accountability starts at the very top, we focus here specifically on health workers and how their performance can be supported by such approaches.

According to behavioural science, we are more likely to practice a desired behaviour if it is easy, if it attracts us, if other people expect us to do it (social norms) and will think poorly of us if we fail, and if it is a “default”, for example more people will donate organs with an “opt out” system than with “opt in”. Contrary to common belief, knowledge and attitude are not good predictors of practice. Despite being aware of guidelines, health providers might not rigorously follow them, possibly with good reason.

One analyst coined the term “mindlines” – which is to say that rather than following technical guidelines literally, what providers do in practice is usually derived from a combination of their knowledge of technical guidelines, their own personal experience implementing those guidelines (which parts had gone well and what not so well), and what their peers do. So providers in fact relied on their own personal algorithms in many situations. Similarly, other behaviouralists have used frameworks borrowed from the criminal justice lexicon to think about “means, motive and opportunity” in understanding health providers’ behaviour.

The “means” aspect translates essentially to whether the provider has gained the right competencies, through training, coaching, practice and feedback.

The “opportunity” aspect translate to other health systems readiness factors that need to be in place for a health worker to do his or her job well - whether there are supplies, fuel for outreach vehicles, good data to guide scheduling and so forth.

The “motive” aspect relates more to behaviours and norms. People function within networks and have different influencers in their lives. Considering the individual political economy (or circle of influence) in which frontline health providers operate is extremely important. While an externally driven programme can make an action easy, attractive and even incentivised, there also needs to be a normative expectation of behaviours within the provider’s immediate network. This is why the buy-in from health departments, the ultimate employer of health workers, as well as the involvement of communities in which they live and operate, is so important. The natural desire of health workers to want to do their job well and be recognised for this by their peers can be one of the most powerful enablers to be harnessed.

Review of the most promising examples of scalable innovation suggests factors in success relate to the extent solutions are integrated into existing systems, institutional, behavioural and political contexts and resource environments. This suggests that a broader, multifaceted, health systems approach to performance is needed.

Applying a health systems lens, the starting point should be not whether a particular solution “works”, but rather asking what is the particular challenge, what are the causes, and what can be done about it. This way of thinking would facilitate selection of innovative digital tools only if they are appropriate for identified challenges.6

We consider below the evidence base and examples of various approaches that impact performance, including learning strategies, performance strategies such as collaborative quality improvement and supportive supervision, and wider health systems levers such as payment mechanisms, data for accountability and recognition, and community oversight.

2.2 Evidence from Systematic Reviews

The Health Care Provider Performance Review (HCPPR)7 by Alexander Rowe et al is the most comprehensive systematic review of strategies to improve health-care provider performance in LMICs that has been done. It selected 670 reports from 337 studies of 118 strategies in 64 countries Its objective was to assess the effectiveness of all strategies to improve health-care provider performance outcomes in LMICs. This section draws heavily on the results of this review and follow up analysis. Overall, this evidence review found that few strategies are clearly effective across the board; results vary widely and there is a great deal of heterogeneity in contexts and implementation quality

The review found that multifaceted strategies targeting infrastructure, supervision, other management techniques, training and group problem solving tended to have large effects Combining training and supervision had larger effects than use of either strategy alone. Group problem solving alone or as part of training showed large improvements in percentage

6 See also WHO Digital Implementation Investment Guide https://www.who.int/publications/i/item/9789240010567

7 Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review. Alexander K Rowe, Samantha Y Rowe, David H Peters, Kathleen A Holloway*, John Chalker, Dennis Ross-Degnan. Lancet Glob Health 2018; 6: e1163–75. Published Online October 8, 2018 http://dx.doi.org/10.1016/S2214-109X(18)30398-X outcomes. Some specific group problem solving strategies such as “collaborative improvement” work well when a network of facilities come together as a cluster (e.g. facilities in geographical area or of same facility type) and collaborate to solve a problem. However the HCPPR analysis suggests that strategies such as group problem solving might be more effective in areas with higher levels of resources than in low-resource settings, i.e. hospitals in low-income countries and areas in middle-income countries that were not entirely rural. Although reasons for this are not entirely understood, this could possibly relate to a critical mass needed of team members available to collaborate on a QI strategy. The influence of context on strategy effectiveness is unsurprisingly highly significant, hence the importance of considering context and then trialling, monitoring and adapting strategies.

For professional health-care providers (generally, facility-based health workers), the effects were near zero for only providing printed information or job aids or Information and Communication Technology (ICT) alone, suggesting that such approaches work best as a complement to other approaches

In Bihar, India, a digital Continuum of Care Services (CCS) intervention was implemented by CARE as part of the Ananya program This involved the provision of mobile-phone-based tools for frontline workers (FLWs) that aimed to increase the coverage and quality of services that FLWs provide and facilitate supervision. The tools combined registration of beneficiaries, scheduling of home visits, and guided protocols along with audio-visual job aids. FLWs received reminders about the timing of home visits and the tool included checklists of information to gather from and provide to beneficiaries during home visits.

A study8 found that FLWs’ understanding of the CCS tool increased but not until the second year of implementation and as a result of an intensive training effort by CARE. Formal training consisted of 16 sessions of approximately 3 hours each held over 8 weeks, as well as informal mentoring involving CARE staff visiting FLWs who were identified as having difficulty. More literate and younger FLWs were significantly more likely to understand the tool. This suggests that hand-holding support needed when introducing new systems needs to be prolonged and not under-estimated.

An important finding was that the level of understanding by FLW supervisors was more limited with about half the supervisors surveyed at endline unable to log into their phone or open the records of home visits conducted by the FLWs they were supervising. FLW reports did not suggest any substantial improvements in supervision of FLWs outside of sub-centre meetings, which was one of the aims of providing the supervisory mobile tool to the ANM.

Impact of the assessment across various tasks was mixed, but effect sizes were generally low, in single figures. A plausible explanation is the lack of clear expectation set by supervisors, many of who themselves did not engage with the tool.

The scope of potential ICT approaches is, however, impossibly wide and there has been further work on refining the definition and typologies covered by this (e.g. by WHO) so that these can be assessed in a more nuanced way.

For lay health workers (typically Community Health Workers), training tended to have fairly small average effects. Strategies that included community support plus training were more likely to lead to larger improvements, although the evidence is limited.

Many effect sizes were less than 30 percentage points, which means that even after implementing improvement strategies, important performance gaps will probably remain. The authors recommend the need for continual monitoring of strategies, so that managers can know how well a strategy is working, address gaps (which are to be expected) by modifying or abandoning the strategy or layering on a new one, and continue to monitor and modify as needed.

Overall, the systematic review emphasises the need for researchers to use better methods to study the effectiveness of interventions. Given what is argued above about the need for iterative improvements where approaches can adapt, layer new elements and improve, it is particularly important that any evaluations allow for a dynamic learning approach rather than a rigid intervention design.

2.3 Training attributes associated with effectiveness

Given that important questions remain about how well training works and the best ways to design training, the systematic review authors undertook further secondary analysis to characterize the effectiveness of training strategies and identify attributes associated with improving provider practices in LMICs.9

This analysis found that educational outreach visits at health workers’ sites were more effective than inservice training, which was more effective than peerto-peer training and self-study. The effect of inservice training when some or all training was done at the health workers’ worksite was 6.0–10.4 %points greater than when all training was done offsite. In-service training that incorporated clinical practice tended to be more effective than training without it, by 6.9–7.4 %-points. Interestingly, five eligible studies of training strategies to improve CHW practices found essentially no effect for in-service training and educational outreach visits.

Analysis found that educational outreach visits at health workers’ sites were more effective than in-service training, which was more effective than peer-to-peer training and selfstudy.

In terms of longevity of training impact, the average effect of in-service training, when done in isolation, declined with time since training, with training effectiveness waning to zero after around 21 months on average. Importantly, when training was combined with supervision, the mean effect did not tend to decrease over time.

Finally, lower baseline performance was associated with greater response to training; for every 10 %-point decrease in baseline performance level, mean in-service training effectiveness was 1.1–1.5 %-points higher.

While these results suggested that certain approaches were more effective, the variability of results and the overall low-quality of evidence suggest that (as the larger HCPPR emphasized) programmes should monitor performance to understand the effect of a given approach in their specific context.

These findings from Rowe et al match findings from an earlier review from 2013 of effective inservice training10 which found that use of multiple techniques that allow for interaction and enable learners to process and apply information work best. Case-based learning, clinical simulations, practice and feedback were identified as effective educational techniques. Didactic techniques that involve passive instruction, such as reading or lecture, were found to have little or no impact on learning outcomes. Repetitive interventions, rather than single interventions, were shown to be superior for learning outcomes. Settings similar to the workplace improved

9 The effectiveness of training strategies to improve health care provider practices in low- and middle-income countries. Rowe AK, et al. BMJ Global Health 2021;6:e003229. doi:10.1136/bmjgh-2020-003229

10 Bluestone, J., Johnson, P., Fullerton, J. et al. Effective in-service training design and delivery: evidence from an integrative literature review. Hum Resour Health 11, 51 (2013). https://doi.org/10.1186/1478-4491-11-51 skill acquisition and performance. This review also found that computer-based learning could be equally or more effective than live instruction and more cost efficient if effective techniques were used.

The above findings are also very much consistent with what is known about the science of how adults learn. In 1984, Knowles devised his 4 principles of andragogy –the method and practice of teaching adult learners – see Figure.

A presentation11 at the Teach to Reach Summit, Seattle, Washington, in November 2015 identified attributes of successful training, some of which particularly relate to these four principles and are highlighted below:

• Training uses stories, case studies, problem-based learning, or simulations

• Trainees are asked to discuss, debate, collaborate or teach each other

• Trainers provide informational feedback (i.e., rather than only praise or criticism) – what was particularly good about the demonstration (or what could be improved and how)

• Trainers help trainees tie the training’s objective to a self-relevant, self-transcendent purpose (e.g., for training on treating an illness, trainers helped trainees understand that improving treatment practices will both make them a better, more respected health worker and save the lives of people in their community)

• Trainers recognise trainees’ growing competence and help trainees develop selfefficacy

• Trainers ask trainees to make a plan on how the new knowledge would be put to use. The plan includes setting goals that are short-terms, specific, and moderately challenging.

Competence can be regarded as a combination of knowledge, skill and attitude which translates to performance at a required standard. It is important to distinguish between the learning required to complete a specific task versus developing competency to perform the full spectrum of a health worker’s or a manager’s role. For example an individual task might be to perform a skin pinch to evaluate dehydration in a child with diarrhoea; but health providers need to be able to carry out the entire set of desired practices needed to evaluate and treat a child with diarrhoea

In the workplace, on-the-job ‘just in time’ learning can be done quite effectively via a reference card or a video, for example, while developing competency as an immunization manager would be a longer process that would require a different approach. Core competencies as an immunization manager might include not only skills in providing vaccination and managing vaccine logistics but also capacity to use data for local decision-making, to analyze one’s own data, and to apply behavioral and social science thinking to partner with communities and clients.

The ultimate measure of training effectiveness is “learning transfer” – the extent to which learning is translated to improved practices in the workplace. Most interventions, if they measure it at all, measure only learners’ perceptions of transfer. A recent review12 attempted to distil validated transfer factors that increase the likelihood of learning transfer, and to make these insights practical for trainers, learning architects, and eLearning developers. This review found that learners who acquire practical skills during training will be more successful in transfer, and that learning skills seems more potent than just learning concepts Learners are more likely to achieve transfer success if they have early opportunities to apply what they have learned in their work. This is because learners will quickly forget what they have learnt without practice, and can also lose confidence and motivation over time. Measurement of impact is further discussed in section 3.9.

2.4 Evidence on CHW Training

A 2018 systematic review13 of ongoing training for Community Health Workers in low-income and middle-income countries suggested that in-service / refresher training was the most neglected and most variable type of CHW training and highlighted the issues of proliferation of different training providers in this space. Interesting, the review found that the majority of ongoing training was delivered in person, with only four studies reporting the use of mobile technologies in training delivery and the majority taking place face to face.

This review suggested that regular in-service training rather than pre-service training was vital, and that co-designing of training with stakeholders including the intended trainees was important to ensure the training was relevant. At the same time, use of a shared training package could help standardise and quality assure training content and bring efficiencies from economies of scale. WhatsApp groups were found to be used effectively for messaging, support supervision, CPD and team-building.

Positive aspects of such social media included its accessibility at times to suit the user, updates, use of a standard and familiar App rather than a bespoke one, ability to substitute for a great amount of face-to-face contact, and that it was useable on users’ own phones rather than requiring laptops or tablets. Downsides were the cost of data and connectivity limitations.14

A study of CHW training programmes in sub-Saharan Africa and South Asia (supported by various mobile telecoms companies as well as pharmaceutical companies and the mHealth Alliance) concluded that whether face-to-face or remote, effective teaching should be interactive, not simply didactic/transmission of information; that smartphones could be used effectively as job aids, with practical application while the CHW was actually working, and that feedback loops were vital for useful learning.15

Sending text messages to CHW's phones to improve competency has been one of the most evaluated interventions and shows a point improvement in outcomes ranging from 0%-24% for child health interventions reported by different country studies.16 Regarding training using traditional face-to-face training, meanwhile, a study of community health volunteers in rural

12 Factors That Support Training Transfer: A Brief Synopsis of the Transfer Research. Will Thalheimer, PhD. Work-Learning Research, Inc. January 2020

13 O’Donovan J, O’Donovan C, Kuhn I, et al Ongoing training of community health workers in low-income and middleincome countries: a systematic scoping review of the literature. BMJ Open 2018;8:e021467. doi:10.1136/bmjopen2017-021467

14 O’Donovan J, O’Donovan C, Kuhn I, et al. Ongoing training of community health workers in low-income and middle-income countries: a systematic scoping review of the literature. BMJ Open. 2018;8:e021467. doi:10.1136/bmjopen-2017-021467

15 The Current State of CHW Training Programs in Sub-Saharan Africa and South Asia: What We Know, What We Don’t Know, and What We Need to Do. One Million Community Health Workers Campaign & mPowering Frontline Health Workers. July 2014

16 Rowe Alexander et al. Improving health worker performance: an ongoing challenge for meeting the sustainable development goals BMJ 2018; 362 :k2813

Uganda found a key requirement was that such training needs to have regular, frequent and local meetings in order for it to be effective.17

2.5 Evidence on Digital Approaches to Supporting Healthcare Providers

A systematic review on the effectiveness of mobile health (mHealth) technologies to train healthcare professionals in developing countries18 was done in 2015. Only seven eligible studies related to medical education in developing countries were identified. Three studies assessed the integration of mHealth solutions into the training of allied healthcare professionals, three assessed resident doctors, and another assessed undergraduate medical students. Six of the seven studies used mobile phones as the intervention tool. The majority of studies indicated that mHealth was a promising tool for education and training of healthcare professionals, although definite conclusions were limited due to the study sizes and quality.

In 2016 Johns Hopkins University were commissioned by the Gates Foundation to undertake a review of Mobile Technology in Support of Frontline Health Workers.19 The report summarised current data from over 140 FHW-supported mHealth projects in developing countries, highlighting the emergent trends and best practices in the use of mobile phones, tablets, and technical platforms by FHWs over the last decade. It detailed the key considerations in choosing the type of phone and platform and associated programmatic costs, presented the evidence on the effectiveness of mobile approaches, and established a framework for systematically deploying such tools. They reported a number of key findings, many of which remain relevant 5 years on:

• Projects primarily supported CHWs and facility staff through facilitating electronic decision support or data collection activities.

• 74% of respondents reported that mobile devices were provided to FHWs by the projects

• Functional requirements for a platform included SMS functionality, ability to work in a low bandwidth or offline environment, ability to create reports and dashboards, and the ability to design workflows. Other broad requirements included open source platforms, ease of use, low costs, interoperability and ease of customization.

• Several popular and established platforms (such as CommCare, Oppia Mobile, RapidPro, OpenSRP that are still widely used today) were used to support a range of functions and are interoperable to some degree. Across the board, however, interoperability remained a challenge for mHealth interventions, with little data on the use of standards for data architecture and interoperability for interventions.

• Just over 10% of the projects recorded in the database used custom-made or proprietary software.

• Evidence suggested that there was potential for cost-savings in the long-run resulting from increasing system efficiency, use of open source technical platforms and content, discounted bulk purchases of equipment and phone services (e.g. minutes, SMS, data plant etc.). They also flagged, however, the lack of consistency in how costs are recorded thereby compromising a rigorous cost effectiveness analysis.

More recently, the prevailing wisdom that there is a lack of evidence on digital learning strategies has been challenged, with over 2,500 RCTs identified in a WHO commissioned

Community

5:62.doi: 10.3389/fpubh.2017.00062

18 systematic review of digital learning.20 WHO has published a discussion paper “Digital education for building health workforce capacity”21 on which many key stakeholders collaborated. This confirms evidence pointing to digital education being at least as effective as traditional education in improving health professionals’ knowledge, skills, satisfaction with the educational approach and professional attitudes.

This paper suggests that offline digital education may improve educational outcomes such as knowledge of trainees and that significant levels of communication competencies can be delivered via digital education, through online digital education, virtual reality and virtual patients.22 The paper also argues that digital education for health workers can be used as an adjunct to traditional education, usually in a blended learning model. It acknowledges, however, that evidence is often inconclusive, for instance, “it appears that end-to-end digital education is more effective in improving knowledge, whereas blended learning is more effective in improving skills, but this also depends on the learning objectives or assessment methods”.

Evidence suggests, for example, that online digital education already plays a notable role in training medical doctors, significantly improving learning outcomes compared with self-directed or face-to-face learning. In the case of antibiotic management, digital education was shown to improve the prescription behaviours of practising physicians better than those who undertook only traditional learning. Blended learning may be more suitable for health care training, however, which commonly needs to combine practical hands on, skill-based training with knowledge of the theory.

One point of note that the paper makes is that most studies (90%) of digital education (of RCT standard) have targeted doctors and much fewer target allied health professionals/frontline health workers such as nurses or pharmacists. It suggests some balancing needed to target more digital education initiatives at these other cadres.

2.6 Specific digital learning approaches

Learning approaches can be broadly classified into traditional, digital and blended approaches Digital approaches can in turn be broken down into off-line digital training (materials pre-loaded onto SD card, USB etc and distributed), on-line training (which can still have offline capability), Massive Open Online courses (MOOCs) and other forms of mobile learning. There are also newer and emerging technologies/approaches including virtual reality, augmented reality and serious games – these are covered in section 2.6

Certainly, the rapid emergence of new technology along with high ownership of mobile phones and expansion of 3G and 4G connectivity in LMICs paves the way for greater inclusion of digital either as stand-alone or as part of blended learning approaches. Selection of media needs to be compatible with the communications infrastructure and the real costs and ability of the intended trainees to access the training – as a one-way process or interactively.

The ability to update, correct or supplement information which is found to be insufficient or incorrect, is vital. Media such as online courses or apps, which allow rapid revision or update, are thus valuable. This benefit has to be balanced against the barriers posed by the cost and practicalities of network capacity of connectivity, compared to distributing content in advance, in fixed form (e.g. in print, or recorded onto SD card, USB, or downloaded and installed onto devices). The larger the scale of the programme, the greater the importance of getting this balance right. In many cases, a combination of fixed content pre-distributed, complemented by

20 Personal communication with Josip Car, Digital Health Education Collaborative from Imperial in Singapore / WHO Collaborating Centre on Digital Health, 25th January 2021

21 Digital education for building health workforce capacity. Geneva: World Health Organization; 2020. Licence: CC BY-NC-SA 3.0 IGO.

22 real-time or periodic updates, is best. This may be achieved by use of messaging channels such as WhatsApp, by face-to-face training sessions, by broadcast or email updates.

2019;21(7):e14676 (http://www.ncbi.nlm.nih.gov/pubmed/31267981, accessed 22 November 2019).

The unit (per learner/trainee) cost of developing and producing courses and training modules, and then of running the training, can vary a great deal according to the sophistication and requirements of the media packages used, the number of trainees, how often it can be repeated, and whether or not the costs of providing hardware or paying for connections has to be covered. These factors are context-specific and need to be considered in programme needs and feasibility analysis, planning and design.

Table 1 presents a summary of Digital Education Modalities along with advantages and constraints making them more or less suitable in different settings.

Table 1. Digital Education Modalities (adapted from Table 3.1 Digital education modalities and definitions in WHO Digital Education for Building Workforce Capacity

Digital education modality

Description

Offline training An intervention that requires no internet or local area network connection and can be delivered through external media including CD-ROM, external hard disc and USB stick

Online training An intervention that requires the use of a “transmission control protocol” (TCP) and an “internet protocol” (IP) as a standard for the learning activities; also referred to as “online”, “web-based” or “networked”. Can also have offline applicability. An example is cloud-syncable courses offered offline through platforms such as the Moodle open-source learning management system

Interactive Voice Response (IVR)

Audio Job Aides

Can work by a “pull” mechanism, available on demand where users dial a short code to hear message. Or delivered through “push” mechanism where health workers receive calls at scheduled time to listen to short modules.

Advantages

Constraints

- Does not depend on connectivity

- Cost-effective to produce and distribute content, especially if health workers use own devices

- Advantage of high quality, pre-prepared content

- Easy to update content

- If offline functionality, users can download and interact offline

- All online learning is scalable – more users at reducing marginal cost

- More difficult to update content quickly

- Expensive if programme needs to supply devices

- High data use requirements

- Users care about storage space on their phones

- Rarely accessible to those in more remote/ impoverished areas without access to internet or smart devices

Decision Support

Applications

A form of eLearning delivered in several seconds

- Good ownership of regular mobile phones and reasonable 2G coverage.

- Works on basic as well as smart phones

- Calls can be free or subsidised – Telecomm /mobile providers incentivised to participate to increase market share

- User friendly translation of clinic or management protocols into advice at point of care

- Easily scalable

- Easy to update content

- Good for settings with high rates of mobile internet penetration

- Users often need credit on their phones to receive calls even when free

- Low pick up rates for “push” calls (though higher when supervisors engaged)

- Can be language barriers in multilingual countries and regions

- Guidance is standardised/binary and might not address the specific problem a provider faces

- Lack of interoperability between different apps and systems. Can result in parallel systems increasing burden on healthcare workers

- Providers often unwilling to log into separate systems

Classified as Internal

Serious games and gamification

Serious games are games designed specifically for the “serious” purpose of providing health professional education via a digital. Gamification is “the application of the characteristics and benefits of games to real world processes or problems”.

- Strengthens learning experience by introducing enjoyment, competition and problem-solving

- Novel experience increases engagement and attention

- Some games require expensive console / PC equipment

- May be less applicable to engage senior staff

- Lack of evidence in LMIC settings

Massive open online course (MOOC)

An online course that is designed for the participation of large numbers of geographically dispersed students.

Virtual Reality VR is a technology that allows the user to explore and manipulate computer-generated real or artificial three-dimensional (3D) multimedia sensory environments in real time. It allows for a first-person active learning experience through different levels of immersion.

Virtual patient (VP)

Interactive computer simulations of real-life clinical scenarios for the purpose of medical training, education or assessment

- Represent a disruptive vision to get learning out to the masses

- Affordable, scalable, and easy to update content

- Newer versions emphasise need for MOOC in combination with other learner support

- Time efficient and cost effective in the long run

- Realistic, repetitive practice allows for an enhanced learning experience

- Potentially replicates positive attributes of training in user’s own work setting and need for incorporation of clinical practice

- Enables clinicians to continuously practice skills and techniques

- Standardised, safe and reduces reliance of patient availability

- Have been high drop-out rates

- Some MOOCs, especially older versions, were technology driven but weak on pedagogy

- Requires access to computing devices

- High set up costs to roll out equipment and scale up in low- and middle-income settings

Mobile learning (m-learning) e.g. WhatsApp groups

Learning across multiple contexts, through social and content interactions, using portable, networked devices

- Rich learning and problem-solving potential through real time, two-way communication

- Social media platforms, such as WhatsApp, are commonly used for communication globally

- Affordable data bundles increasingly provided by Telecomm/mobile providers specifically for WhatsApp use

- Quite complex to design and set up to achieve quality learning.

- Requires local context adaptation and validation

- Users need to use their own data bundle which can result in increased personal costs to the user

- Numerous messages could evoke user fatigue

- Encroaches on personal life and could impact wellbeing

- Only available on smartphones

2.7 Performance Theory

For decades, researchers have developed and built on models which seek to explain human performance and how this is best shaped. The Behavior Engineering Model (BEM) developed by Thomas Gilbert23 is fundamental to many later models and provides a way of systematically identifying barriers to individual and organisational performance.24

Roger Chevalier updated and adapted this model using language commonly employed in describing worker performance.

As with the original BEM, it focuses on the distinction between environmental and individual factors, with environmental factors as the starting point because, in the words of Geary Rummler and Alan Brache, “If you pit a good performer against a bad system, the system will win almost every time”.25

In the article “Training Alone is not Enough”, 26 the authors present findings from IntraHealth’s research in Human Performance Technology (HPT) on the factors influencing family planning provider performance. The study drew on earlier human performance models (developed by Rummler & Brache in 1993; and by Stolovitch & Keeps in 1999) that assumed that clear expectations, timely performance feedback, adequate environment, incentives (and consequences), and skills and knowledge are required for a worker to perform well. The study explored how each of these conditions affected healthcare provider performance in four countries.

This study showed significant association between the performance of workers and conditions in the workers’ environment. When viewed across all countries, non-monetary incentives were the most powerful “predictor’ of performance when compared with other factors while knowledge and skills were found to be the least influential factor overall.

These findings are highly significant given the dominance of training as a relatively expensive solution and the concurrent under emphasis on good management practices - which include performance reviews to set clear expectations, feedback and verbal recognition - as well as approaches which s increased respect from both the community and the supervisor will have zero or small financial implications but a potentially large impact.

The following sections look at some of the practical ways a performance culture can be established.

2.8 Collaborative Quality Improvement

Among performance management strategies, collaborative quality improvement (CQIs) have been used to improve health care for several decades. CQIs involve the use of healthcare teams from different sites to improve performance on a specific topic by collecting data and testing ideas with improvement cycles which can be supported by coaching and learning sessions. The district level of the health system is well positioned to facilitate systematic group learning among facilities of similar types and across tiers of the health system. District-led area-

23 Human Competence: Engineering Worthy Performance. Thomas Gilbert, 1978

24 Updating the Behavior Engineering Model. Roger Chevalier, CPT Performance Improvement, v42 n5 May-Jun 2003 http://www.ispi.org/publications/pitocs/piMay2003.htm/

25 Ibid citing Rummler & Brache, 1995

26 Training Alone Is Not Enough.- Factors that Influence the Performance of Healthcare Providers in Armenia, Bangladesh, Bolivia, and Nigeria. Lauren Crigler et al. Performance Improvement Quarterly, 19(1) PP 99-116 based learning and planning bring together providers and administrators responsible for a catchment area to solve clinical and system problems, harmonize approaches, maximize often limited resources and create better communication and referral between facilities.27

Evidence shows that CQI when supplemented with health worker training result in positive patient outcomes for formal health providers. There is less known about the positive health impact from such approaches in the context of lay health workers.28

The following example is drawn from a successful trial of a CQI in Niger and Mali.29

Example 1: CQI to address Post-partum Care in Niger and Mali

A large-scale programme in Niger and Mali sought to improve postpartum care for mothers and newborns. The approach involved identification of the root causes of failures to complete all steps in the active management of 3rd stage of labour. Facilities collaborated in identifying and trialling solutions to the specific micro-issues identified, for example the fact that although oxytocin injection was indicated within 1 minute of birth, this rarely happened in time. New solutions were trialled, for example keeping a pre-prepared loaded syringe in an ice-bucket next to the delivery bed. Importantly, data was collected and used by the facilities themselves at very regular intervals on where measures introduced were having effects. As a result of this improvement effort, local ownership and shared learning to accelerate implementation of best practices, 78 facilities demonstrated rapid improvement incompliance with standards for post-partum haemorrhage prevention and essential newborn care as well as a reduction in estimated postpartum haemorrhage. The case study found that this approach yields rapid results and can be efficiently spread to improve care in low-resource settings.

Similarly, a research study in South Africa concluded that an approach of continuous quality improvement (CQI) based on continuous mentoring of CHWs by their supervisors was found to improve the CHWs’ delivery of services to pregnant women and mothers in both quantitative and qualitative terms. The CQI followed an initial two-week face-to-face training in WHO’s Community Case Management. The study concluded that training using low-tech, face-to-face mentoring, with printed support materials, could be effective, on the basis that it was carried out locally and with frequent, regular mentor-trainee interactions built into the CHW’s annual calendar.30

2.9 Supportive Supervision

The effect of supportive supervision strategies on health worker practices in LMICs is highlighted by another review which identified attributes associated with impactful supervision.31 Provision of supervision to supervisors and supervisors’ engagement in problem solving with healthcare providers, were two features strongly associated with a positive effect on provider

27 Garcia-Elorrio E, Rowe SY, Teijeiro ME, Ciapponi A, Rowe AK (2019) The effectiveness of the quality improvement collaborative strategy in low- and middle-income countries: A systematic review and meta-analysis. PLoS ONE 14(10): e0221919. https://doi.org/10.1371/journal.pone.0221919

28 Ibid

29 Improving postpartum care for mothers and newborns in Niger and Mali: a case study of an integrated maternal and newborn improvement programme. M Boucar et al. USAID Applying Science to Strengthen Improve Systems University Research Co., LLC 18 September 2014

30 Horwood et al. Human Resources for Health (2017) 15:39. DOI 10.1186/s12960-017-0210-7

31 Samantha Y. Rowe, PhD*; Dennis Ross-Degnan, ScD; David H. Peters, DrPH; Kathleen A. Holloway, PhD; Alexander K. Rowe, MD. The effectiveness of supervision strategies to improve health care provider practices in low- and middle-income countries, 2020 (draft, under publication) practices, from an analysis of 81 studies across 36 countries. Training of supervisors, provision of supervisory checklists, and frequency of supervisory visits were not meaningfully associated with health worker practices.

Previously, hierarchical models of supervision emphasising inspection and control have been promoted, but more collaborative supervisory strategies are now widely advocated. Such strategies might typically involve record reviews, observations, performance monitoring, constructive feedback, provider participation, coaching, problem solving, and focused education.

A study from 201632 looked back over nationally representative surveys of health systems in seven countries in sub-Saharan Africa and pooled these with clinical observations to examine the association of in-service training and supervision, both popular strategies, with provider quality. The results showed that observed quality of care was poor, and associations between quality and in-service training and supervision were modest although there were marked variations between countries. However, at most, improvements related to interventions were equivalent to 2 additional provider actions out of the 18–40 actions expected per visit.

This study also found that effective supervision must include problem solving in the context of an ongoing supervisory relationship.33 The authors suggest promising strategies such as addressing supply shortages in facilities in conjunction with coaching providers; more effective use of data feedback for health workers combined with action plans detailing appropriate responses to challenges in providing patient care, and engagement and capacity building of health system managers to identify poor quality and take actions to address it.

The role of mobile phones to strengthen supportive supervision for CHWs was studied in Kenya. A WhatsApp group to facilitate instant messaging was created for CHWs and their supervisors to ‘support supervision, professional development, and team building’. Importantly, the authors of this study reported on the quality assurance and information exchange, which the system facilitated, and on the supportive environment fostered by the use of the technology.34

2.10 Payment for performance

Payment for Performance (P4P) - There is a considerable literature on this but much is dated ten to fifteen years ago and interest seems to have peaked. There is considerable evidence from the hospital sector in all countries that provider payment mechanisms can affect performance positively or negatively. In the UK the capitation system for primary care is adjusted to reward key targets including immunisation coverage and appears to be effective in that targets are met. However there has not been a control study. In primary care in LMICs there is no clear evidence that P4P works. One reason is there is not a standard blueprint for P4P schemes being used in LMICs. A recent scoping review identified 41 P4P schemes in 29 LMICs. This variation may well reflect the fact that P4P schemes have been tailored to meet certain policy objectives and respond to different conditions on the ground. The review could not identify common features that would affect results.

A useful review concludes the evidence on P4P in LMICs is still in its infancy, both in terms of evidence of impact (especially as far as health outcomes are concerned), and in terms of the attention to potential unintended consequences.

32 Training And Supervision Did Not Meaningfully Improve Quality Of Care For Pregnant Women Or Sick Children In Sub-Saharan Africa | Health Affairs. Hannah H. Leslie Anna Gage Humphreys Nsona Lisa R. Hirschhorn Margaret E. Kruk: September 2016 https://DOI.ORG/10.1377/HLTHAFF.2016.0261

33 Measuring the quality of supervisor–provider interactions in health care facilities in Zimbabwe. P Tavrow, YMI Kim, L Malianga - International Journal for Quality …, 2002 - academic.oup.com

34 Henry JV, Winters N, Lakati A, et al. Enhancing the supervision of community health workers with whatsapp mobile messaging: qualitative findings from 2 low-resource settings in Kenya. Glob Health Sci Pract 2016;4:311

25.

One concern with P4P is the potential for gaming: the data used to measure performance might be manipulated by the service provider to inflate reported performance for example for child immunisation, and it can be difficult to verify that the immunisation took place. The ideal response to gaming is for the funder to base the P4P scheme on information which is outside of the control of the delivery organisation and which is easily measurable, so that there is little scope for misreporting. Vaccinator tracking systems using GIS technology and photo upload are proving useful for verification of outreach work.

A second concern is cherry picking. If the cost of providing an incentivised service differs across patients, and differences in costs are not reflected in the P4P payment, the delivery organisation may have a financial incentive to select patients with low cost and avoid patients whose costs are above the tariff e.g., hard to reach children and mothers. Solutions which mitigate this include reimbursing against mileage travelled with GIS verification, so health workers visiting more distant locations for immunisation outreach, for example, are properly compensated.

Overall introducing P4P schemes needs to be approached with considerable caution with the need to ensure the country health system has the resources to design, implement and monitor it effectively. The costs of independent verification can also be disproportionately high, which is why integrated platforms that have a built-in verification function are valuable.

2.11 Data for Accountability and Recognition

Making data public has an impact on performance as this can foster healthy competition, and acts as a disincentive for poor performance that is within the control of the health worker. Frontline health workers often work among rural populations, with only sporadic contact with supervisory staff. Web-based dashboards allow supervisors to track the performance of community health workers individually or at the district/regional/national level, either by noting the volume of digital productivity or by real-time GPS tracking of workers as they perform their field activities. This enables supportive supervision to those workers who may be lagging in their performance, while also enabling the recognition and reward of exceptional field staff. These approaches are embedded within a number of mHealth service packages.

In Pakistan, several different but similar approaches for digital vaccination tracking have been pursued:

Example 2: EPI Program, Punjab Information Technology Board

E-Vaccs digital application scaled out across 3 of 4 provinces of Pakistan designed by the Expanded Program of Immunisation Program in partnership with inhouse IT Punjab Information Technology Board. It features a smartphone application provided to vaccinators which is used to monitor vaccinator attendance, outreach visits through geographical geo-tagging, information feeds on immunisation volume used to construct an e-vaccination card and SMS reminder to children of defaulting parents. Realtime records are streamed through a central database into digital dashboards that are used by the EPI program and senior leadership to review monthly program performance. Evaccs use for immunisation monitoring led to rapid increase in immunisation coverage in Punjab province, attributed to strong government ownership, leadership engagement with immunisation performance monitoring and competition between districts for performance recognition. E-Vaccs has relied mainly on vertical implementation to achieve results and does not have supportive supervision elements, hence with leadership changes in recent years it has faced compliance issue from vaccinators in districts with weak governance.

Zindagi Mehfooz is a somewhat similar phone-based, electronic immunisation registry across all 29 districts of Sindh which records and analyses programme data in real time to produce a variety of performance-based immunisation reports, while Teeku is another (see Example 3)

Example 3: Teeku District pilot, Tando Muhammad Khan, Sindh, Pakistan

A successful multi-faceted and integrated intervention which included digital data for accountability, supportive supervision and payment for performance was the Teeku pilot in Sindh, funded by Gavi and carried out by Aga Khan University with Department of Health Sindh. This included i) Immunization registry: child and household registration with timelines of scheduled dosage by child and village, defaulter alerts and chat support; ii) GIS tracking of vaccinator visits; iii) Micro-planning targets linked to refreshed annual plan, monthly recognition at district immunisation meetings; iv) Pay for performance: vaccinators were provided fuel allowance based on verified visits. Vaccinators reported that the app was user friendly and helped them to track defaulters. Working vigilantly and ‘honestly’ under their supervisors was reported as a new and enjoyable experience. Photo verification of child vaccination was considered to be the best feature, while a video feature helped educate communities.

District managers reported that the App helped in improving vaccinator compliance and that monthly performance review meetings occurred regularly. While the Micro-plan target setting and fuel money provision linked to App helped chase coverage rates, they reported that notice/ salary suspension to non-performing vaccinators did not always help. Although not scaled further, lessons are being applied to 2 more districts in KP and GB regions.

The concept of SMART regulations involves introducing softer regulations, especially selfregulations, alongside traditional sanctions-based regulations. It works of the premise of coregulation whereby regulator, health providers and a range of other stakeholders such as accreditation institutes, insurance schemes and consumers themselves all co-produce desired behaviour in health care providers through a range of regulatory instruments and broadening of regularly responsibility.

In South Africa, the Office of Standards Compliance (OHSC) launched a patient-centred accountability service called ‘Rate my Clinic’. This was a cell phone-based patient rating system developed to score clinics based on user visit experience. Clinic scoring was done on a range of dimensions including staff attitude, infection prevention, cleanliness, waiting times and drug availability. It served as an early warning system for regulators and helped to effectively apply limited inspectorate resources to low scoring clinics. It also strengthened the interface between quality of care and public accountability. This was piloted successfully in one state before being scaled up across South Africa.

2.12 CHW Performance and Community Involvement

More focused evidence on lay workers is available from a systematic review of interventions to improve CHW performance.35 This found that positive changes in community behavioural outcomes and improved use of services can result from CHW interventions if upstream completion of prescribed activities and downstream adherence to protocols are the focus of interventions. Supervision, incentives and equipment/ commodity support are key parameters that influence CHW performance. Incentives work well when CHW have a single repetitive task but less well when they are expected to multi-task.

A qualitative review of factors influencing CHW performance proposes that CHW performance is affected by hardware factors such as logistics, supply, training, supervision, accountability mechanisms and communication support, as well as software factors such as relationships, power, values, norms and interests related to community, CHW program management and other healthcare workers.36 Competency development in soft skills such as communication, confidentiality issues and handling community relations is important and relatively over-looked as compared to hard skills of service delivery. Similarly, the role of non-financial incentives is important for CHW delivery and can take the form of transport support (e.g. bicycles), recognition (e.g. CHW days), preferential access to certain health or educational services etc.

In 2018, WHO, UNICEF and The Lancet produced the first evidence-based global guidelines for health policy and system support to optimise community health worker programmes. According to this document, there was evidence that broad strategies (e g competency-based education, supportive supervision, and payment) are effective. However typically evidence was not sufficiently granular or too context-specific to allow recommendations on specific interventions for example which supervision strategies or which bundle of financial and non-financial incentives were most effective. But involvement of the community in selecting CHWs, observation of service delivery, performance data and community feedback were all found to be effective. The guideline recommended remunerating practising CHWs with a financial package commensurate to the job demands, complexity, number of hours, training, and roles that they undertake, and recommended not paying CHWs exclusively or predominantly according to performance-based incentives

The guideline also suggests consideration be given not only to traditional performance measures, but also to basic labour rights that include safe and decent working conditions and freedom from all kinds of discrimination, coercion, and violence. Some of these aspects are of particular concern and relevance in both acute crises and chronic complex emergencies, as there is a growing body of evidence that CHWs have a strong potential in mitigating their negative health impact.