3 Enabling Principles for Innovative Performance Strategies

This section examines some key principles associated with success of performance strategies derived from the evidence base, and provides examples of these principles being applied in practice as well as explanations for why best practice is not always followed.

Some important principles, many somewhat overlapping, are as follows:

➢ Using a blended learning approach, where components can be layered and tailored to requirements, generally works best

➢ Using different media and formats tailored to context increases success of training

➢ Co-designing with health care providers themselves allows training to focus on practical issues rather than theory, thus becoming more relevant and immediately transferable to health providers’ practice

➢ Including problem-solving approaches and collaborative quality improvement addresses specific identified challenges rather than relying on approaches which deliver standardised content

➢ Group dynamics, peer support, buddies, and communities of practice are an important part of skill-building and norm setting

➢ Ensuring co-production with government is more likely to lead to ownership and government buy-in and sustainability

➢ Linking training with accreditation as part of a lifelong learning approach increases uptake/completion rates and ensures better alignment with national health workforce strategies

➢ The feasibility of technology must be considered though digital approaches add value in fragile settings

➢ Better measurement approaches are needed to support tailored learning

These principles are expanded and illustrated below.

3.1 Using blended and layered approaches

Blended learning is very often the preferred approach and digital interventions that have been used conjunctly with other learning strategies have been more successful than a sole reliance on digitalization. A blended approach makes best use of remote learning through technology, in combination with the richness and close interaction of face-to-face meetings. However, face-toface training may itself be ineffective if not well prepared and not available at the time and place it is needed, or if it is simply transmission of information. It can also be expensive, difficult to achieve to schedule, of low or uncontrolled quality, and disruptive to other commitments.

The remotely-accessed components are of most value for quality of carefully pre-prepared, expert content and learning activities; access at times and in the blocks which are needed by the user at any stage; and for opportunities for recap, for wider peer group interaction and for frequent update.

Dependent on available expertise and budget, interactive, self-check activities and learning games can very usefully be developed and built into the web-based material. Broadcasts on TV or radio, email correspondence, closed chat groups, bulk or individual SMS messaging, individual or group voice calls, printed self-access material can all be useful. Preferably, centrally produced content may be supported and mediated with local practice or workplace application, mentor, tutor or peer group support.

Example 4: RAPID IMMUNIZATION SKILL ENHANCEMENT (RISE) by JSI & Ministry of Health & Family Welfare, India

Rapid Immunization Skill Enhancement (RISE) is a blended package to strengthen the ongoing training of frontline health workers engaged in routine immunization in India with an alternate capacity building model during COVID-19. This user centric model leverages the high technology (mobile phone, internet) penetration among the health workforce and effectively uses a digital training content supported by a face-to-face component of mentoring by supervisors. The digital part is primarily selflearning through an objective focused content based on adult learning principles that engages the learners through a variety of audio, animated visuals, reading and interactions like learning games and quizzes. It also has a provision of objectively assessing the learning and certification to make it more attractive to the learners. An open-source Learning Management System (LMS) provides a realtime monitoring dashboard which enables problem-solving and mentoring by supervisors. The intervention has been rolled out in 5 states of India, is available in 4 regional languages, and has 3,000 health providers to date. Users have fed back that the mobile App could be more user-friendly so that it can be easily operated by all health care workers and that it should be updated frequently as the guidelines in immunisation program changes very often.

Example 5: Community Health Academy: Liberia by Last Mile Health

The Community Health Academy partnership was set up in 2017 to deliver educational text, video, and quiz content to first level health workers in Liberia through an integrated training/ support application. 100% of government FLHWs were equipped with digital aids and assessments indicate increase in prenatal visits and correct diagnosis/ referral of malaria cases by health workers.

Liberia’s limited telecommunications infrastructure compounded by dense tropical rainforest and frequent rain means connectivity is often sporadic and unreliable. To address these challenges, the Ministry of Health distributed new curriculum using Bluetooth capabilities. The Liberian experience has been drawn on by Harvard University to develop a global blended learning curriculum for CHWs. This is designed to be easily adapted to local contexts and offered as a public good for ministries of health worldwide to adapt, aligned to WHO guidelines, protocols and global standards for quality.

Blended and layered approaches are inherently more complex than simpler, single modality interventions and complexity makes rolling out at scale more difficult. “Cascade approaches” to training are common, where master trainers train others to deliver the training at lower tiers of the system. While this system has advantages of greater coverage at lower cost, quality can be dissipated, and there might be more tendency to stick with a training scheme that people are familiar with.

In this situation, however, a mix of digital and face to face delivery would often be optimal, as digital delivery would allow more standardised, quality-assured delivery, while involving actors from different tiers of the system to deliver training can increase their ownership of the process.

When designing brand new training schemes, it is easier to incorporate digital as part of a blended approach from the start.

Example 6: Surge vaccinators, UK

In the UK, the National Health Service (NHS) has brought in St John Ambulance and the Royal Voluntary Service to coordinate the training and deployment of thousands of volunteers to help ensure the safe and smooth running of vaccination services. Volunteers are first screened via on-line application, identity, and police record check and by an on-line interview. Volunteers are trained using a blended learning approach. They first complete on-line training, which can be completed in around 3 days, and need to pass each module before proceeding to the next. Training then culminates in a one-day face to face classroom training, where participants undertake role plays to practice various techniques under different scenarios. They practice giving an injection, injecting water filled syringes into silicon bags strapped onto the arms of fellow volunteers.

3.2 Using different media and formats tailored to context

Remote technologies themselves are in general most effective and reliable when used in combinations, rather than solely relying on a single medium in isolation. Media choice may include web-based learning material in audio, video, text and visual graphics.

Films can be used within traditional and digital training settings, and are very effective to create an emotional engagement with the subject matter which supports retention of the message and internalisation of the learning. Films are often valued by communities in unstable and volatile countries, providing women and their families with accessible, relevant and potentially lifesaving knowledge. Films made by Medical Aid films in South Sudan to increase awareness amongst women of the benefits of proper nutrition and correct pre and postnatal care were very popular.

Short films and even text messages to health workers can also be used to reinforce health workers’ credibility and the validity of their advice to communities.

Example 7: Medical Aid Films

Medical Aid Films has produced over 300 films which are viewed 500 million times per year. Their films have enjoyed success in low resource settings, used for example in Somaliland as part of elearning for the Nurse and Midwife Association to show techniques to be practiced in parallel in the classroom such as resuscitation on a model baby. Community workers from partner Africa Education Trust report that the films are encouraging positive changes in the health practices of local women and girls. The films are also popular amongst men, with some men and adolescent boys even taking part in subsequent discussion sessions, and also used in facility waiting rooms to encourage impromptu discussions. An excellent film has been produced on vaccine hesitancy, with support from the London School of Hygiene and Tropical Medicine. A key enabler is the ability to translate content to local languages and mix in local footage along with standardised content.

Tablet based training based on video modules using scripts and films, followed by skills-based questions was provided to nurses for improving competency to provide lifelong ART therapy to HIV positive pregnant women. Nurses found e-training easier to understand and more engaging than usual face to face paper-based learning, and resulted in increased knowledge competency. A key enabler was appropriate local content of videos and engaging adaptation, helped by formative assessment for design and use of local theatre groups for adaptation.

37 Rassi C, Gore-Langton GR, Gigudu Walimba B, Strachan CE, King R, Basharat S et al (2018) Improving health worker performance through text messaging: A mixed methods evaluation of a pilot intervention designed to increase coverage of intermittent preventive treatment of malaria in pregnancy in West Nile, Uganda. PLoS ONE 13(9): e0203554.

3.3 Designing with user needs in mind

It is noteworthy that in a review of ongoing training of CHWs in LMICs, only 5 out of 35 studies reviewed documented seeking input from CHWs as part of qualitative research before design. A major reason for dissatisfaction with much conventional training is inadequately qualified trainers who are unfamiliar with providers’ working environment and ground realities. Given the lack of documented participant input and feedback in terms of programme design, delivery and evaluation, this review suggested greater use of participatory action research (PAR) – i.e. working ‘with’ end-users in a collaborative effort rather than ‘for’ or ‘on’ them 38

There is of course a natural tension between the need not to “reinvent the wheel” and the need to design specific to the user’s context. There might well be existing products that can be adapted, but the real need for adaptation might not be obvious until the tool is properly fieldtested. That is when an organisational inertia to making changes and adapting can set in, particularly when programmes are up against delivery targets. While the phrase adaptive management is much talked about, what it really entails in practice is perhaps under-estimated.

In response to demand for remote training on COVID-19, UNICEF were pushed into a new space and collaborated with the Digital Classroom initiative to adapt their content into a training module for FLWs, initially in Liberia, Chad, Togo and DRC, and now expanding to 8 further countries. The intervention was tested in a pilot with 30 CHWs in Liberia and content was offered across three platforms: regular SMS, Moodle and the Internet of Good Things (IOGT). The preferred platform was SMS due to CHWs’ existing ease and familiarity with this, with Moodle coming second, which had the advantage of allowing multi-media content.

Example 9: Immunization Academy Watch

IAW is a global initiative providing rapid access to short, “just in time” practical videos for task-based immunisation training and delivery in the field. It uses online videos and resource documents accessed through their website as well as an app for smart phones. It covers a wide range of immunisation topics as well as new content on working safely during the COVID-19 pandemic, and has a WhatsApp group of users. Content was developed through in-depth consultation with user groups in Tanzania and combined with existing immunisation “best practices” from WHO. Effectiveness is measured through knowledge increase measured by quizzes, and by the growing number of users, particularly in countries like Nigeria, India and Pakistan. There has not, as yet, been any attempt to assess providers’ competence to carry out the task being trained following exposure to the videos. This initiative is sponsored by a grant from the Bill & Melinda Gates Foundation and led by learning experts at Bull City Learning. For now the only model is donor funding as user populations cannot fund it directly, although a country subscription model for future would provide a more sustainable funding source.

38 O’Donovan J, O’Donovan C, Kuhn I, et al. Ongoing training of community health workers in low-income and middleincome countries: a systematic scoping review of the literature. BMJ Open 2018;8:e021467. doi:10.1136/bmjopen2017-021467

Example 10: Village Reach, with Praekelt and Viamo

The partners provide remote, phone-based training to more than 100,000 health workers through a Gavi INFUSE partnership. Praekelt combines two proven solutions together: leveraging the established MomConnect platform and combining with health Hotlines, using WhatsApp for one-toone engagement, targeting both health users and health workers. For health messaging, they offer flexibility so users can choose the best method to reach (whether via smart phone, IVR, or the ability to talk to a person).

Connectivity is an issue and some health workers formed their own groups to listen together to get around this problem (although this made it difficult to count how many had done the training). A module on cold chain maintenance involved a video (using the ECHO platform) but ran into issues with bandwidth. Low pick up rates were an issue, and although free of cost, IVR calls might not connect if a participant has no phone credit. Mobile network operators are providing competitive data bundles for people to engage through WhatsApp. Praekelt has been able to lean on mobile providers to secure good rates although said it was difficult to reach an agreement.

Providers were reported to miss in person interactions, while an important missing link for training was follow up supervision. Remote training is a good way to reinforce simple messages as part of a ‘package’.

3.4 Incorporating problem-solving approaches

A common theme from the evidence is that to be effective, training and supervision need to address specific identified challenges rather than relying on standardised, content-driven approaches. Problem solving has been shown to increase the effectiveness of training and supportive supervision.

WhatsApp has taken off worldwide partly because it is so accessible – mobile operators are now providing competitive data bundles for people so they can engage through WhatsApp. Another important reason for WhatsApp’s popularity is that it allows two-way, real-time communication and short concise answers to day to day issues that arise.

There is currently great interest in the potential of more democratic “bottom up” approaches to sidestep traditional power hierarchies and vested interests in the traditional “training economy”. So-called Distributed Networks can crowd source solutions to problems by seeking help from peers facing similar challenges, even if working in different locations. Such approaches seek to empower health workers to apply their local knowledge to solve problems.

The Geneva Learning Foundation (TGLF) is the best example of such an approach in the immunisation context (see example 11). Their aim is to contribute to transforming learning, leadership, and training to strengthen immunisation programmes. The approach draws on George Siemens’ Connectivism learning theory which emphasises the importance of networks, self-organization (the spontaneous formation of well-organized structures, patterns, or behaviours), and lessons from chaos theory about complex patterns and effects of small changes in initial conditions that are important properties of learning and decision-making.

TGLF’s approach plays on the moral dimension and reinforcement of positive norms –connecting health providers with other providers who are motivated to do their jobs well and achieve results. Their model pays no per diems or other incentives and expected rewards are derived from doing a meaningful, valuable job well and achieving better results.

There is potential for organisations such as Gavi to explore how learning from this newer innovative model can support its growth, and how action plans identified through selfestablished, voluntary networks can feed into health systems strengthening plans supported at country level, especially where challenges come down to lack of resources for implementation.

Example 11: Geneva Learning Foundation, Teach to Reach Accelerator Conference

TGLF’s approach aims to make peer monitoring and support into a scalable system. A Teach to Reach Accelerator Conference was held in January 2021, aimed at country-based participants from its network of over 25,000 people. The approach uses a collaborative problem-solving approach to immunisation challenges which values the inherent knowledge, expertise and experience of health workers about their own working context. Participants prepare action plans, which are peer-reviewed, to deliver on actions in line with country plans. 549 Teach to Reach Alumni collaborated in an intensive three-day Immunisation Training Challenge Hackathon (ITCH) to problem-solve their immunisation training challenges in October 2020. Over 4000 immunisation professionals have joined the Scholar COVID-19 Peer Hub.

In July 2020, 500 meetings were held by scholars in countries with no travel or per diems involved, which is indicative of the power of distributed networks as a force for change. Online social networking enables communities of practice that offer potential to bring together temporally and geographically dispersed actors to work towards a common purpose. There is however a risk of lack of coordination and alignment with government plans and specific country performance objectives.

3.5 Communities of practice and peer support

Communities of practice provide a knowledge-sharing platform for peer learning. Virtual communities of practice can be created for case-based learning through use of tele-mentoring, case studies and guides, and multi-directional learning communication. The success of virtual communities of practice is dependent on basic infrastructure such as electricity and internet bandwidth, use of cloud-based videoconferencing platforms through a webcam-enabled computer, tablet, or smartphone, contextual adaptation of case material and familiarity/ comfort of local users and local experts to participate

Virtual communities of practice such as those that use the Project ECHO model (Example 12) go beyond MOOC-style virtual classrooms by developing knowledge networks that promote real-time multidirectional learning and teaching, with a strong emphasis on peer-to-peer sharing and learning. Knowledge delivery and mentoring are combined with opportunities for live discussion and problem solving in situations where meeting in person is impractical and costprohibitive.

Challenges to the implementation of virtual communities of practice include the need for clinical and public health experts and participants to have protected time to participate, and adapting a model of learning that is non-hierarchical, participatory, and dynamic in contexts where such an approach might be unfamiliar.

Example 12: Extension for Community Healthcare Outcomes (ECHO)

Project ECHO is not directed specifically towards frontline workers but is increasingly being adapted to support community-health-worker initiatives. The ECHO Model can be defined as tele-mentoring, a guided practice model where participants are part of a knowledge-sharing platform through virtual communities of practice and case-based learning. It uses multipoint video conferencing that connects specialist teams at local, regional, and international academic medical centres and centres of excellence with primary care teams and community health workers in rural and underserved locales. The ECHO Institute works with partners to support a global COVID-19 response focusing on IPC, care and management of patients, as well as other diseases and health initiatives.

Virtual communities of practice such as Project ECHO goes beyond virtual classrooms by developing knowledge networks that promote real-time multidirectional learning and teaching, with a strong emphasis on peer-to peer sharing and learning. Knowledge delivery and mentoring are combined with opportunities for live discussion and problem solving in situations where meeting in person is impractical and cost-prohibitive. The success of virtual communities of practice is dependent on basic infrastructure such as electricity and internet bandwidth. However, cloud-based videoconferencing platforms are now accessible to nearly anyone with a webcam-enabled computer, tablet, or smartphone.

3.6 Co-production and government buy-in

Ensuring government ownership is a prerequisite to sustainability but can be challenging because of a lack of capacity within ministries, especially in digital technology. Constraints on government spending budgets also hinders efforts to support introduction of new systems while turnover within government departments imparts more challenge.

Decentralised systems (or dysfunctional centralised system) can make obtaining buy-in tricky, as a ministry of health at central level might approve a solution while provinces take their own view and require different sign offs. In DRC, approval processes for interventions have sometimes become stuck at province level. In Somalia/Somaliland, the different ministries of health often want their own systems, each with its own version of DHIS2 for instance. We are aware of instances where a Government has signed an MOU to take a solution to scale and then asked for a kick-back from the agency to proceed, which, when declined, meant the government went with a different solution.

Where the demand originates from government, and fills a recognised need, there is evidently greater buy-in. In Malawi, Village Reach was asked by the College of Health Sciences to develop remote training specifically for a new Pharmacy Assistants Training programme. This cadre existed on paper but not in reality. Village Reach worked with Government from the beginning to establish a suitable pre-training course which is being continued by the College. While it is not yet fully funded by the Government, it is ostensibly owned by the government and likely to be sustained.

Example 13: MOTS - Mobile Training & Support Service, Sierra Leone

Implemented by the public-private partnership EBODAC (Johnson and Johnson, World Vision, Grameen Foundation) and the Ministry of Health & Sanitation (MoHS). MoHS in Sierra Leone has developed, tested and rolled out an innovative way to provide refresher trainings to a large group of remotely located CHWs. IVR was selected as the technology to deliver audio-based refresher trainings on the topics of vaccines and outbreak response including Ebola disease surveillance procedures. MOTS was developed in close collaboration with MoHS. Training content was customized in line with the national training curriculum and case reporting requirements. The technology was found to be readily accepted by the CHWs and their engagement was such that they also provided important elements to be improved prior to further implementation. There was however signs of general fatigue of the IVR methodology for participating in the quiz assessments which required further investigation. It was found that repetitive aspects of important training content can be reinforced without the need for additional classroom presence of the CHW community. Sustainability requires cost containment and subsequent software accessibility for authorities. Transparent partnership and alignment with the MoHS from the outset of this project is considered an important element of its success.

Example 14: Government-branded App ‘COVID-19 Ethiopia App’

Last Mile Health (LMH) has supported the Government of Ethiopia to produce the first MoH digital learning platform attracting many users and reaching beyond the expected numbers and areas. The App consists of 6 modules to train a broad range of frontline health workers and inform others about COVID-19. The GoE see it as a launching pad for a further culture of digital learning in health. Because LMH had an existing relationship with MoH to develop technology-based digital learning platform for RMNCH, it could rapidly switch to developing an COVID app due to urgency. Among reported challenges, only 6% of users completed all 6 modules, possibly due to incompatible Android versions. 87% of App users are male, whereas 53% of health workers in Ethiopia are female, but mostly residing in rural areas and without access to internet. The App has been heavily promoted through Facebook which is used by more men than women in Ethiopia.

While the need for country buy-in is self-evident, it can be difficult in practice for a combination of reasons. Ministries are large and complex organisms and support from one quarter does not signify support across the board. Information may not flow well within a ministry or there might be internal tensions or power struggles. High turnover can also hinder government engagement. The bandwidth of busy department officials is a major issue and development partners reported difficulty in being able to access and engage government counterparts. Most development partners do take great pains to ensure government approval or sign off, which is often a formal requirement, but there is a great difference between sign off and true ownership and the solution being actively supported by a country government. Genuine engagement requires time and patience for relationship-building and to co-create workable solutions that a ministry will want to get behind.

3.7 Recognition, accreditation and lifelong learning

There is a close link between government buy-in and government health systems formally recognising and accrediting training within their health workforce development strategies. The next section discusses further the importance of expectations of those in positions of influence on providers’ behaviour. Drop-out rates of online training can be high but are much lower when training is a formal requirement of continuing certification or career development.

“BMJ Learning” is accredited in 70 countries and confirms that linking training with accreditation for continuing medical education or continuing professional development improves uptake. They report current direction of travel is to link continuing CME and accreditation. While this experience may not be directly applicable to front line workers, getting some sort of formal accreditation or recognition should help uptake.

WHO conducted an internal review and launched the vision of WHO Academy representing a complete shift from a standardised, content-driven training approach, where everyone gets the same course, to a lifelong learning strategy, enabling tailored learning to different users to acquire competency and make behavioural changes. The vision of the new WHO Academy is that regulators worldwide will formally recognise a WHO certificate. A more globally joined up approach to accreditation would certainly help portability of health workers’ qualifications.

Accreditation itself does not have an impact unless there are human resources policies that recognize and reward increased competencies. Accreditation with measurable indicators that are assessed by an independent body, similar to accreditation of academic programs, can increase stakeholder confidence in the quality of a training program. Further, if a certificate of completion of the accredited program is formally recognized as a means for career advancement and is funded as a line item in a MOH budget, quality and sustainability are more likely to be achieved. Ethiopia is a good example of a government which has worked to establish a career ladder for its CHWs (Health Extension Workers) to improve motivation and retention and in turn quality of care.

One reason that in-service training is not always linked to formal accreditation, is that it is often times driven by ministries of health or development partners who liaise with their health sector counterparts whereas the higher educational institutions that provide pre-service qualifications to the health workforce might come under the education sector, a different set of stakeholders. Bringing the two closer together to promote a lifelong learning approach, will involve closer multi-sectoral coordination by governments as well as development partners.

Example 15: LEAP (Amref) Kenya

Leap! Is a public private partnership between Accenture, M-Pesa Foundation, Safaricom, Vodafone (Mezzanine) and the Government of Kenya which has trained over 35,000 learners across 30 counties in Kenya. This is a pilot initiative that seeks to assess the ability to enhance routine Immunization and defaulter tracing training by integrating an mHealth component that offers initial and refresher training on Immunization to Community Health Volunteers (CHVs), collaboration and supervision opportunities to improve effectiveness of the program. Leap! offers accredited healthcare training content and is customisable to needs. Operating principles include aligning fully with Ministries of Health in Africa in the empowerment of their health workforce and developing Africa’s first fully integrated mobile health platform. It is currently embarking on scaling up across various countries in Africa aiming to be the "goto" solution for health worker training in Africa and beyond.

Example 16: Health[e]Foundation

HealtheFoundation provides Continuous Medical Education for HCWs covering topics such as HIV/AIDS, TB, mental health, basic occupational health, research, health problems among refugees, as well as preventive programs on sexual and reproductive health. Programs start with a kick-off workshop, followed by a self-study period, and a follow-up workshop, but can be adjusted to meet users’ needs. The e-course curriculum can be accommodated on any global open-source platform, as the e-modules are developed using the universal SCORM format. They can be accessed online via a website, offline on a USB stick and via an App. 20,000 health care workers have been trained in 34 countries. In March 2019, 110 participants from Nigeria finished HIV training - 90% passed and received certificates co-signed by the University of Amsterdam Academic Medical Centre.

3.8 Feasibility / Acceptability of Technology

The following lessons have emerged on the relevance, feasibility and acceptability of technologies commonly used in digital training or data exchange.

Offline capability is essential as most settings experience inconsistent connectivity

It is often now the norm for systems to have offline functionality so that content can be downloaded in advance and not rely on consistent connectivity. Remote training works best when delivered in short bites, while bookmarking technology which enables users to return to the course where they left off is also importance for user ease.

Many apps work offline once downloaded, although the amount of storage capacity apps take up on personal devices can be a concern. Bluetooth data transfer is also being used in low connectivity environments, for example when supervisors visit remote locations and can collect data from health workers via Bluetooth.

Telehealth models cannot work offline and a good internet connection is needed for video conferencing. The success of virtual communities of practice is dependent on basic infrastructure such as electricity and internet bandwidth. However, cloud-based videoconferencing platforms are now accessible to nearly anyone with a webcam-enabled computer, tablet, or smartphone, and the growth of cellular availability in sub-Saharan Africa and lowincome countries has enabled many new applications of technology in health.

Providers prefer to access content on their own devices rather than carry separate devices

There is a strong preference for content to be watched on providers’ own devices and this avoids escalating costs of providing hardware as programmes are scaled up. This has been reported from both low and high resource settings.

The British Journal of Medicine (BMJ) who provides online training (BMJ Learning) and decision support apps (BMJ Best Practice) has found that if people can access training on their own devices, they are more likely to use it as doctors do not want to carry two phones. Also, attention spans are short and the shorter the chunk of learning the better.39

Where tablets are provided by programmes, aside from the cost, there can also be logistical hurdles as devices are sometimes collected at the end of each day due to high risk of theft. As ownership of smart phones increases, we would expect more reliance on personal devices.

However, there are of course many settings where video content is desired and health providers do not have smart phones. A low-cost Android tablet, connecTAB, has been used to deliver video tutorials and remote online peer-tutoring for clinical skills training. The connecTAB, which is significantly less expensive than internet-enabled smartphones, was specifically developed for areas with low bandwidth. Videos can be preloaded in order to circumvent the issue of streaming in areas with slow internet speeds, which is a common barrier.

Example 17: Low-cost

tablets and videos to teach clinical skills in Kenya

ConnecTAB, a low-cost Android tablet, was used as an educational aid for teaching clinical examination skills amongst medical students in Western Kenya. Observed Structured Clinical Examination (OSCE) scores were compared between a cohort of students who had connecTABs and a cohort of students who did not. The videos were developed by Geeky Medics, an organisation specialising in open-access clinical video tutorials. Results showed significantly higher improvements in the scores for both cardiovascular and abdominal examinations within the group who received the e-tablets as compared to the control group. The study suggested that access to connecTAB improves clinical education and efficacy and holds promise for international training in both medical and allied healthcare professional spheres in resource-limited settings.

Mobile phone training through IVR can be an effective complement to classroom training in CHWs, even in fragile states

Use of voice recording is often preferred to written content particularly in low resource settings and can be well tailored to local settings. In India, BBC Media Action found only 9% of CHWs had ever sent an SMS. Recognising that they needed something simple and audio based, they created a suite of mobile health services using interactive voice response (IVR) This technology is handset independent, audio based and accessed via a simple voice call.

Viamo has developed IVR based solutions, known as its 3-2-1 service, in a growing number of countries through partnerships with mobile network operators (MNOs). These MNOs provide free airtime in return for securing loyalty to their sim card brand. IVR can work on a “pull” basis, where users dial a short code to hear a message, or on a “push” basis when health workers are called on the phones at a set time, which might be repeated as necessary for missed calls. In DRC Viamo found pick up rates rather low (30-40%), though these improved greatly when supervisors engaged and reinforced the need for training. There is a risk of general fatigue of the IVR methodology and for providers to remain engaged, hence the importance of a blended approach which involve some face to face sessions.

To use IVR optimally, recordings should be developed using iterative user-centred design processes, to consider desired speed of delivery, local dialect, rich idiomatic phrasing to ensure memorability, and emotional as well as rational arguments to make content appealing and persuasive. IVR benefits from multiple rounds of user-tests to check and finetune content.

Multi-function systems which allow functional integration are far more likely to be used

Busy providers do not want to have to log in and out of different systems to perform functions. The extent to which eLearning is integrated into other digital systems e.g. electronic health records (EHR), is of major importance and attracting growing attention. Providers are more likely to use a clinical pathway decision support app, for example, if they can link directly to this when they are with a patient and completing an electronic health record and want to check the correct tests they should be requesting. These need to be granular links which go directly to the relevant piece for clinical workflow. Integration is in fact essential for the benefits of Artificial Intelligence and Machine Learning to be realised.

It is also common sense that a wider multi-function platform that health care workers are already familiar with will be easier and more likely to be used. Familiarity and frequency of use increases providers’ confidence and skills navigating a new system which can otherwise be daunting and off-putting. Medic Mobile’s Community Health Toolkit (CHTK) provides an "end to end” community health systems platform – incorporating messaging, digital decision support, data collection, task and schedule management, patient files and analytics.

Sehat Kahani is an existing mobile app used in India which allows healthcare providers to evaluate, diagnose and treat patients via chat, audio or video. During the COVID pandemic, content has been added on to the existing app, building its potential as a wider multi-function platform that users and health care workers are familiar with. Similarly, the mobile and web application, ImTeCHO in India had a range of functions but is also used to pay performancebased payments through the App, obviously encouraging engagement and uptake of the app.

Support for change management and technological troubleshooting is vital to allow transfer of ownership

Transitioning technological solutions to new owners needs time for them to bed in. Lessons from Medic Mobile, Acasus and others is that handing over responsibility for digital maintenance is best done gradually, with troubleshooting support along the way. South African digital technology social enterprise firm, Jembi, commented that digital systems are “easy to build” but the hard graft is to manage implementation and the change management needed on the ground. Jembi itself has now stopped taking on projects without a clear sustainability plan

The NGO Village Reach has a Transition strategy with two key staff positions responsible for managing transition activities with government and relevant partners.

Living Goods, describing itself as “technology agnostic”, helps governments and implementing partners select, design, configure, and implement digital solutions that best meet local needs. Understanding that technology alone is not sufficient to deliver health outcomes, it works to ensure ‘wrap-around’ services to build the processes and systems to ensure technology is repaired and maintained, that users adopt and own the system, and that the data is accurate and verifiable

Example 17: Living Goods and Medic Mobile, Kenya and Uganda

Living Goods and Medic Mobile have collaborated to create a set of mobile and web tools to support CHWs. The open-source Smart Health app is used by CHWs to support delivery of high quality and integrated primary health care services, including advice on COVID-19. This application also has a supervisor dashboard that enables the real-time remote management of CHWs, manages effective stocking of medical commodities, and is being used to provide the government with critical data to better plan and budget for community-level interventions.

Living Goods was involved in co-designing the solution with technology partner Medic Mobile, who has now fully handed over the technological management to Living Goods. A lesson is that handing over digital technology capability should be done gradually with partners, and build up capacity incrementally, rather than handing over everything in one go as there is always

Example 18: Living Goods and Medic Mobile, Kenya and Uganda

Living Goods and Medic Mobile have collaborated to create a set of mobile and web tools to support CHWs. The open-source Smart Health app is used by CHWs to support delivery of high quality and integrated primary health care services, including advice on COVID-19. This application also has a supervisor dashboard that enables the real-time remote management of CHWs, manages effective stocking of medical commodities, and is being used to provide the government with critical data to better plan and budget for community-level interventions. Living Goods was involved in co-designing the solution with technology partner Medic Mobile, who has now fully handed over the technological management to Living Goods. A lesson is that handing over digital technology capability should be done gradually with partners, and build up capacity incrementally, rather than handing over everything in one go as there is always troubleshooting to be done and time is needed to build up these skills.

Technology helplines have been used effectively to provide support in case of any technology related problem reported by individual users. In the ImTeCHO example in India, a lesson was that ICT support was critical for sustained support on mobile maintenance and application updates to field-based health workers. A simple, intuitive and voice navigated user interface was important for uptake, while ASHAs also needed regular supervisory feedback to ensure high and effective use of the application. Trouble shooting technology issues in the formative phase, involving ASHAs in iterative solving of App features and coaching sessions with ASHA workers for hands on support helped to ensure satisfactory uptake and adherence.

Building modular content using existing “global good” platforms supports integration, interoperability and sustainability

There are various open-source platforms that have become well-established – such as OpenSRP, CommCare, Community Health Toolkit to mention just a few – and powerful new technology is continuously emerging and evolving. Many of these platforms are designed to be interoperable with other parts of the system. OpenSRP, for example, can complement and add value to other “global good” digital health information systems, including medical records systems (OpenMRS), health management information systems (DHIS2), logistics information systems (OpenLMIS), and messaging platforms (RapidPro) that are deployed at scale. Annex 5 provides a summary of the main platforms with potential for further exploitation for immunization performance tracking.

Key informants reported that it is “technically not that challenging” to make various systems interoperable, which in many cases is preferable to the alternative of dismantling functioning systems in some parts of the country in favour of a single national digital platform. Where workflows and dashboards are developed in a modular way, they can be more easily adapted to existing platforms. Ministries are more likely to need help with configuring and deploying the most appropriate solutions in the field, rather than creation of new systems.

TIP Global Health (formerly The Ihangane Project) has developed E-Heza, Rwanda’s first pointof-care digital health record which utilises real time data trends to tailor health education to individual families and to improve health care delivery. TIP is working with the Rwanda Ministry of Health to integrate E-Heza into their data-reporting systems (DHIS2 and OpenMRS) while also adding additional functions. The aim is to bring E-Heza to all health centres and CHWs in Rwanda by 2021 and all of East Africa by 2023.

Example 19: Terre des Hommes, Burkina Faso

An App – leDA - co-created by Terre des Hommes (TdH) and the MoH in Burkina Faso has digitalised the WHO medical protocol Integrated Management of Childhood Illness (IMCI) and guides health personnel to diagnose sick children accurately. Data is analysed to improve the quality of care and inform decision-makers. The digital tool is built on Dimagi’s CommCare platform. Using an existing digital platform allowed Tdh to focus on the design of the digital tool rather than the platform itself, and comply with individual health data collection, transmission and storage regulations. A LSHTM evaluation found a large effect on % IMCI tasks done correctly but found no effect on improving the % of children with medicines correctly prescribed. This underscores the need for monitoring to allow continuous improvement of interventions so that they can achieve impact in saving lives.

TdH is preparing the field so that the Ministry of Health of Burkina Faso could take full control of the digital solution IeDA and directly manage it, with IeDA deployed in 67 per cent of all health centres in Burkina Faso, in two health districts in Mali and in Niger, by end of 2020.

Open source is desirable but does not mean solutions become “free”

In LMICs, use of open source software (OSS) is preferred over proprietary software, for reasons of ongoing costs of licencing, adaptability and scalability. There is a misperception, however, that OSS systems are free in the longer term, when in fact the cost of the license is a very small component of the total cost of ownership of the solution and long term sustainability needs to be financed. Some countries have developed a digital health investment case to guide the development of digital health systems and motivate for funding from the national treasury for identified technologies and systems with proven healthcare benefits.

There is a need for international standards

Standards for software development (interoperability, reusability, accessibility, manageability and durability) need to be considered, with implications for protecting and maximising value obtained from investment in developing e-learning. This issue has been driven by US agencies – including USAID and BMGF – and also increasingly by the UK.

Since 2005, the South African digital social enterprise firm, Jembi, has been helping low resource countries develop country variants of standards, working with WHO. South Africa has been at the forefront in terms of pushing for interoperable standards and there is a new National Health Normative Standards Framework for Interoperability in eHealth in South Africa (HNSF) reflecting international best practice

FHIR (Fast Health Interoperability Resources) HL75 standards are emerging as the most important international standard These started to be introduced over the past 5-6 years in a slow process, building on work started with development of the Principles for Digital Development, by USAID, UN and the Rockefeller Foundation. One of FHIR’s goals is to facilitate interoperation between legacy health care systems, to make it easy to provide health care information to health care providers and individuals on a wide variety of devices from computers to tablets to cell phones, and to allow third-party application developers to provide medical applications which can be easily integrated into existing systems.

FHIR provides an alternative to document-centric approaches by directly exposing discrete data elements as services. For example, basic elements of healthcare like patients, admissions, diagnostic reports and medications can each be retrieved and manipulated via their own resource URLs.

3.9 Measurement, Individual learning and Incentivisation

The issue of measurement of training impact is widely seen as a neglected area in need of attention. It is essential for course correction, ensuring strategies can be optimally tailored to context, for ensuring strategies translate ultimately to high quality patient-centred care, and for ensuring cost-effectiveness of investments.

Much training is delivered by turning technical guidance into presentations, with everyone getting exactly the same content delivered over several hours or days. Learners with different needs and learning pace are thus lumped together, with the result that individual learning needs are often not met While personalised learning is seen as priority, the challenge is delivering this at scale. Some digital education has been conceptually pedagogically weak, with more attention to the technology than learning impact. More than ever, pedagogical experts and software developers need to work closely together, to ensure that the power of digital and data analytics can be used to accurately pick up individual weaknesses or difficulties and support tailored learning. This includes greater sensitivity of measurement with feedback loops and remedial action to ensure required competencies and learning needs are met. However current measurement practices are often limited to counting the number of training events and number of people trained, with less attention to how formal training translates into performance. Where they do exist, the most commonly available measures of impact are improved knowledge scores, measured by pre- and post-test scores. Such tests are not really proof of training impact - anyone taking the same test twice is likely to do better a second timewhile increases in knowledge do not necessarily translate to improved performance. While more significant than knowledge-only measures, expanded coverage is at best an indirect measure of improved health provider performance.

Directly observing competency is a more rigorous approach though requires more resources. Objective structured clinical checklist examination (OSCE) is a type of examination used in medicine to test clinical skill performance and competence in a range of skills, and serves as a practical, real-world approach to learning and assessment.

Example 20: One Million Lives Initiative. Maternity Foundation and Laerdal Global Health

The Maternity Foundation’s One Million Lives initiative - Helping Babies Survive, Helping Babies Breathe, and Helping Mothers Survive (supported by Laerdal Global Health with American Academy of Paediatrics and Jhpiego) uses a blended learning approach backed up by standardised competency assessment. During 2020 they began to turn face to face courses into an online platform which could be run anywhere in the world. As the skills being taught require hands on practice, training is a hybrid of virtual and face to face, with the trainer joining on-line but trainees assembled in groups in remote settings. The group dynamic is important as the peer-to-peer methodology involves learners practising skills together until they have mastered a skill. The OSCE style assessment has been digitised. An app shows the procedure and a checklist and learners can practice until competent and confident. Learning is measured and there is now a focus to move towards analysing data from the training to see which HCPs are learning well and which need more support.

Observing performance in a field setting is more logistically challenging. Many organisations use a structured checklist as part of supportive supervision (although interestingly evidence reviews have found mixed results on the effectiveness of such checklists). The utility of checklists likely depends on what is done with the data and on the quality of interaction between supervisor and supervisee. Ideally, data from field supervision is used to prioritise those in need of more support and identify areas for targeted support.

For non-clinical skills such as health programme design or management, it is less clear what is the practical skill to be practised and observed. During training, learners could be tasked with simulation exercises, to produce a micro-plan for a vaccination campaign, for example. However the ultimate measure of training impact will be to monitor empirically what happened following training using simple, SMART indicators.

Example 21: PSI’s Health Network Quality Improvement System (HNQIS)

The INGO Population Services International developed a Health Network Quality Improvement System (HNQIS), a scorecard to track improvements in terms of observed skills according to defined standards of care. HNQIS is an electronic tablet-based application composed of four modules that support healthcare supervisors to: (1) plan supervision visits, using a prioritization matrix that presents facility-specific quality scores and patient volume, (2) assess providers’ quality of care against clinical standards, (3) improve providers’ quality of care through tailored feedback, and (4) monitor quality improvements over time.

HNQIS works offline, so consistent connectivity is not required. Quality scores from HNQIS assessments for family planning in Cambodia, Kenya, Mali, Uganda, and Zimbabwe from 2016 to 2018 showed improvements over time. In Somalia, PSI produced digitized checklists approved by MoH. Provider-client interactions were observed and improvements recorded.1

Incentivisation

Training budgets, as commented earlier, continue to absorb a lot of available funding. Paying participants a per diem to attend training is common, based on the rationale that health workers tend to be low paid and should not be left out of pocket from attending training. However per diems have become a de facto salary top up in many settings, often subverting the motivation to attend training to something financially-driven. Per diems have become so entrenched, that where some face to face training is replaced by virtual learning with no per diems, there is resistance. Viamo reported such resistance as a cultural hurdle when rolling out remote training via IVR in DRC.

In the context of low resource settings and constraints on budgets to adequately pay the health workforce, there is certainly an argument for ODA funds being channelled to frontline health workers if this can leverage quality. Not all development funds can be spent on short term “investments”, as the real challenge comes in sustaining implementation and quality delivery. The problem is that currently paying for training or paying per diems simply incentivises attendance at training and does not differentiate according to impact on performance.

The Framework for Immunisation Training and Learning (FITL) is a shared conceptual framework developed by a range of individuals convened under the Teach to Reach initiative by the BMGF to create a better environment for training and learning. This framework stresses both measurement and feedback of both formal and on-the-job learning, to allow for continued improvement, and at the same time better coordination and alignment of funding and incentives. This Framework emphasises Individual Learning Journeys as key to supporting the needs of managers and the health workforce Whether referred to as user-centric learning, tailored learning or individual learning journeys, these concepts are gaining a lot of traction. Strengthening individual learning involves fostering, recognising, and rewarding self-directed learning in the workplace, as well as offering post-training, on-the-job support to encourage and reward retention and programme improvement.

Other than directly observing performance, either during training or in the field by a supervisor, another option is to obtain feedback from users of a health system. Given the importance of health systems delivering high quality services, which defines quality to include the user experience, capturing these perspectives could be an important part of measuring performance, along with other types of data. This might be especially useful for measuring provider attitude or behaviour given the inherent bias when he or she is observed. Digital solutions that exist for training also offer potential for soliciting community feedback. For example, Viamo’s 3-2-1 platform is already used for survey data collection. If needed, respondents can be incentivised to take part with a small amount of phone credit, and this type of data collection is very costeffective and scalable. The anonymity of a mobile phone response can also solicit more honest feedback with less risk of courtesy bias. There are many other firms – GeoPoll, 60 Decibels and more – specialising in rapid cost-effective real time data collection to support agile and adaptive programming.

Example 22: MomConnect for user feedback, Praekelt

MomConnect is a multi-faceted programme that creates demand for maternal health services as well as improves the supply and quality of those services. It includes stage-based health messages developed by the National Department of Health with support from its partners at Baby Center, a text-based helpdesk that provides answers to pressing questions, a library of health information accessed via a USSD menu, and a service rating feature that allows subscribers to report on the quality of services they receive at facilities. Integrating directly with the national health system – with partners at Jembi and HISP – MomConnect presents decision-makers with control interfaces and dashboards to ensure that feedback from mothers reaches the highest levels of government.

With diverse methods available to measure training impact and provider performance – from structured clinical assessment, field observation, community feedback, as well as HMIS data –there is scope to replace some of the current training expenses - costly hotel, catering and per diems - with alternative funding channelled to frontline health providers based on transparent, data-driven performance measures. Indeed, where training pivots from being mostly classroombased to greater reliance on digital channels, there are savings which could be distributed evenly. A win/win scenario would be for health providers to enjoy benefits in terms of more efficient use of their time and less time spent away from their homes, with an opportunity to earn “top-up” payment based on actual performance.