Toolkit for TPM project by IFRC-MENA

Toolkit for Setting up a Third-Party Monitoring Operation for the COVID-19 Vaccination Roll-out July 2021 Public

List of Acronyms IFRC

International Federation of the Red Cross and Red Crescent

NDV P

National Deployment and Vaccination Plan for COVID-19 Vaccines

Field Monitors

PME R

Planning, Monitoring and Evaluation and Reporting

Information Management

IDP

Internally Displaced Person

World Bank

Third Party Monitoring

Country Office

Health Facility

Job Description

Field Monitor

TPM

Public

WHO

www.ifrc.org

World Health Organization

Table of Contents

Section 1

Background

Section 2

Preparatory Phase

Section 3

Implementation Phase

Section 4

Reporting Phase

Section 8

Further Readings

Section 7

Annexes

Public

www.ifrc.org

Background

Rania Ahmed Deputy Regional Director Middle East and North Africa Regional Office IFRC .

Foreword

COVID-19 has changed the world as we knew it before 2020. More than 200 million people worldwide have contracted the disease and more than 4.4 million lives have been lost to this vicious pandemic. But the pandemic effect on individuals, families and the entire globe goes beyond numbers and figures, with almost everyone on the planet changing the way they live in response. The COVID-19 situation in the MENA Region, was even more challenging with several humanitarian emergencies, protracted crises, natural disasters, and armed conflicts onsetting before and during the global pandemic. Accordingly, the International Federation of Red Cross and Red Crescent Societies (IFRC), alongside the Movement Partners, have been engaged and proactive in addressing community needs during those challenging times, leveraging the National Societies’ auxiliary role to Governments across the MENA Region to support their response to community needs. In that context, and since COVID-19 vaccination national deployment plans were developed, IFRC has been engaged through National Societies on different fronts within those national plans including monitoring the vaccination rollout to ensure quality, safety, and equality. In Lebanon, IFRC was contracted by the World Bank, to conduct a third-party monitoring operation of Lebanon’s COVID-19 vaccination deployment plan and rollout, implemented by the Ministry of Public Health. To initiate this operation, this Toolkit was developed to provide operational and practical guidance on kickstarting a Third-Party Monitoring operation to monitor national deployment plans and roll-out of COVID-19 vaccination. This Toolkit provides a general practical and applicable roadmap, that could be tailored and contextualized to specific needs and requirements among various settings. IFRC MENA Regional Office appreciates and thanks the World Bank, IFRC Lebanon Country Delegation and IFRC Regional Office staff who worked on further enhancing this Toolkit. We also value the work of key experts who provided feedback and comments. The IFRC MENA Regional Office will continue to support National Societies through providing evidence-based approaches and technical guidance prioritizing their needs and development to ensure that National Societies remain to do what they do best, contributing to the development and humanitarian needs of their communities.

Public

www.ifrc.org

Background The Coronavirus disease 2019 (COVID-19) is defined as illness caused by a novel coronavirus called “severe acute respiratory syndrome coronavirus 2” (SARS-CoV-2; formerly called 2019-nCoV), which was first identified amid an outbreak of respiratory an illness cases in Wuhan City, Hubei Province, China1. It was initially reported to the World Health Organization (WHO) on December 31, 2019. On January 30, 2020, the WHO declared the COVID19 outbreak a global health emergency2. Then, On March 11, 2020, the WHO declared COVID-19 a global pandemic, its first such designation since declaring H1N1 influenza a pandemic in 20093. The WHO estimated that most countries globally would start large-scale vaccination efforts in 2021. As countries focus on prioritizing vulnerable populations, it is likely that demand will outpace supply. Current global guidelines call for the prioritization of health workers, the elderly and those suffering from pre-existing conditions, which increases their risk of severe illness. Reaching these people will require new vaccination strategies in most countries, leveraging the experience and capacity of existing programs that focus on childhood and maternal immunization, as well as influenza vaccination5.

The deployment of COVID-19 vaccines in countries around the globe gives rise to several opportunities and risks. It ensures a higher level of protection for those most in need. It also fosters collaboration and coordination amongst different entities, as the National Deployment and Vaccination Plan for COVID-19 Vaccines (NDVP) is often developed by national authorities in close collaboration with relevant development partners. The NDVP encompasses all the key elements recommended by the World Health Organization and represents a central part of countries’ vaccination readiness.

✓ The large-scale acquisition and deployment of COVID-19 vaccines entails risks such as elite capture, fraud and corruption, due to the fact that this unprecedented vaccination effort stretches delivery capacities and given the likelihood of demand outstripping the supply in the early phase of vaccine rollout. Interventions to improve the traceability of COVID-19 vaccines throughout the supply chain and down to the vaccine recipient level will be a necessary mitigation measure.

✓ Institutional capacity for implementation and sustainability. The key institutional risk remains the capacity of the Ministries of Health to carry out the proposed activities. This is heightened by the complexity of vaccine acquisition and subsequent deployment, especially in the absence of adequate funding, as experienced in some countries.

✓ The complexity of vaccine deployment is further exacerbated by the large presence of refugees and internally displaced persons (IDPs) in many countries, creating the potential risk of lack of equitable access to and distribution of vaccines.

Yet the roll-out of the vaccine presents numerous risks, particularly political,, governance, and macro-economic risks. There are also several risks associated with the delivery of COVID-19 vaccination, which include the following:

Public

www.ifrc.org

Background As countries gear up to deploy COVID-19 vaccines, they will need to design and implement monitoring systems to measure the progress and effectiveness of vaccination programs. This involves measuring vaccine uptake and coverage among the overall population, as well as among the at-risk populations prioritized for vaccination.

M&E SYSTEMS

Background

Key Messages DISAGGREGATION

Specifically, the system should allow for disaggregated monitoring among health workers, older people (over 60), people living in at-risk settings such as long-term care facilities, and people with a pre-existing health condition. Furthermore, it will be important to monitor equitable coverage, for example among sexes or across regions within a single country. Targets and population estimates will be needed for priority groups.

EQUITABLE COVERAGE

VACCINE PRODUCTION DIFFERENT UCTS

M & E TOOLS

DATA SHARING

Differences in efficacy, safety and dose requirements will necessitate the monitoring of different vaccine products separately. Ahead of vaccine introduction, countries are urged to develop or adapt monitoring tools and systems, including home-based vaccination records, facility-based vaccination registers, tally sheets, and electronic database systems such as health management information systems (HMIS) and EIR. Countries will be requested to share vaccination, safety, surveillance, and other program data with regional and global level in a more frequent and timely manner than for other vaccination programs. The requirements for regional and global reporting will be kept at a minimum to facilitate high-quality reporting.

Source: Monitoring COVID-19 vaccination INTERIM GUIDANCE COUNTRY READINESS AND DELIVERY Considerations for the collection and use of vaccination data. WHO: March 2021

https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/how-covidspreads.html?CDC_AA_refVal=https%3A%2F%2Fwww.cdc.gov%2Fcoronavirus%2F2019ncov%2Fabout%2Findex.html

https://www.medscape.com/viewarticle/924596 and https://www.nytimes.com/2020/01/30/health/coronavirus-world-health-organization.html

https://www.nytimes.com/2020/03/11/world/coronavirus-news.html#link-682e5b06

WHO SAGE Roadmap for prioritizing uses of COVID-19 vaccines in the context of limited supply https://www.who.int/docs/defaultsource/immunization/sage/covid/sageprioritization-roadmap-covid19-vaccines.pdf, accessed 23 February 2021 Monitoring COVID-19 vaccination INTERIM GUIDANCE COUNTRY READINESS AND DELIVERY Considerations for the collection and use of vaccination data. March 2021.

Monitoring COVID-19 vaccination INTERIM GUIDANCE COUNTRY READINESS AND DELIVERY Considerations for the collection and use of vaccination data. March 2021.

Public

www.ifrc.org

Background Considering the opportunities and risks mentioned above, it is crucial to ensure equity and equality in immunization, and this can be done by setting up a Third-Party Monitoring Mechanism (TPM) involving all the elements of COVID-19 vaccination, which include but not limited to: storage, stock and temperature maintenance across the supply chain, service delivery at vaccination sites, eligibility of vaccine recipients and capturing client perspectives and feedback.

Target Audience

✓ This guidance document is primarily directed at IFRC teams (managers, officers, delegates) responsible for the management, implementation and monitoring of COVID-19 vaccine introduction and delivery in their countries. It could also be useful for partners providing support or organizations developing and deploying information systems to support vaccination programs.

Target Audience

Public

www.ifrc.org

Background The toolkit focuses on the process of setting up a Third-Party Monitoring (TPM) system to monitor vaccination service delivery, including vaccine uptake and service availability, readiness, and quality. It is divided into several sections, each focusing on the different steps required for setting up a TPM system. This toolkit includes different components and aims to provide guidance and tools for all stages of the TPM operation as follows:

Preparatory phase focuses on setting up the methodology, understanding the terms of reference, developing the structure and organizational structure of the TPM operation, developing and adjusting the risk matrix, and, finally, designing the job descriptions of the team.

Implementation phase involves the recruitment of the TPM team, setting and testing the data collection tools, training of field monitors, designing the information flow charts and the data analysis tools.

Evaluation phase covers the final evaluation approach, lessons learned and best practices.

Scope of the Toolkit

Reporting phase focuses on the information flow approach, reporting formats (infographics, dashboard, daily, weekly, and monthly reporting formats) and incident reporting format..

The success of the TPM operation depends on the quality of the data collected. As such, a special section of this toolkit is dedicated to the requirements and topics for the selection and training of Field Monitors.

Public

www.ifrc.org

Background This toolkit was developed based on the experience of IFRC MENA in setting up a COVID19 vaccine deployment TPM operations in Lebanon supported by the World Bank (WB). It also reflects the current understanding of how the best monitoring practices from childhood immunization and influenza programs can be applied to the monitoring of COVID-19 vaccination, as reflected in the WHO Global Guidance.

This toolkit is a “work in progress” which needs to be updated based on the lessons learned from the experience gained within each country. The considerations presented in this document build on the initial recommendations listed in the WHO Guidance on developing a national deployment and vaccination plan for COVID-19 vaccines (see Further Reading). A previously published WHO Regional Office for Europe Operational Guide (see Further Reading) includes a brief presentation of minimum data requirements, discusses the potential use of existing monitoring systems for influenza and childhood immunization, and provides more detailed information on the development of population estimates for priority groups.

Toolkit Development Methodology

Public

www.ifrc.org

Preparatory Phase Preparation is key to setting up a successful and smooth TPM operation. It is recognized that most TPM operations are often set up in emergency settings, i.e. there is a need for quick response and implementation. Nonetheless, this should not diminish the importance of the preparatory phase and assimilation of lessons learned from IFRC TPM operations around the globe. The preparatory phase comprises several steps as follows: Designing the TPM project to ensure that all donor requirements are considered and fulfilled through developing a proposal that includes the following:

A project organizational structure, including workflow and reporting lines;

A detailed timeline project activities plan;

A risk mitigation matrix;

A detailed job descriptions of the team and

A detailed budget

Understanding the Scope of the TPM Operation

Understanding the Scope of the TPM Operation The IFRC team will collaborate closely with the partner entity to ensure clear and mutual understanding of the roles and responsibilities of each party. Each TPM operation will have different objectives and requirements. Therefore, it is critical to develop a common understanding of the objectives and desired outputs of each TPM operation.

Public

www.ifrc.org

Preparatory Phase the TPM operation in Lebanon, the following Scope of Work were agreed upon between the World Bank (WB) and the IFRC: To assess the storage, handling, transportation, and distribution of vaccine supplies in line with WHO guidelines at key points of the supply chain (including but not limited to arrival point in Lebanon prior to customs clearance, centralized storage prior to distribution and vaccination sites).

➢

To ascertain that the number of doses (and vials) consumed daily are in line with the number of beneficiaries covered at that site as well as with acceptable wastage (<5%).

➢

To assess the cold chain management including temperature and stock maintenance of vaccine supplies at the key points of the supply chain.

➢

To assess the delivery of vaccination services at all vaccination sites in terms of processes, site requirements, eligibility of recipients as per the NDVP, adherence to vaccination protocols (including protocols related to eligibility), infection prevention, record keeping and reporting, and waste management plan, especially for waste resulting from vaccination deployment (biological, waste, and other hazardous byproducts that could be injurious to human health).

➢

To assess the perspectives and feedback of vaccine recipients and service providers at all vaccination sites, through grievance reporting mechanisms (GRM) and on social media sites.

➢

To verify the project management team’s knowledge/familiarity and compliance with social and environmental requirements, processes and procedures as defined in the cleared and disclosed Environmental and Social Instruments. This should include but not be limited to: monitoring and documenting registered grievances, including those associated with sexual exploitation and abuse/sexual harassment (SEA/SH), while ensuring confidentiality and anonymity; transparency in the communication of eligibility criteria and deployment; adherence of different project actors to codes of conduct; implementation of COVID-19 preventive and social distancing measures at vaccination sites; assessment of occupational health and safety procedures and infection control strategies put in place to mitigate vaccination deployment hazards;

➢

To develop and propose corrective measures to the WB team, as needed, to address instances of non-compliance or to enhance environmental, health, safety, and social performance.

➢

To share findings and suggested recommendations for action with the World Bank and the Joint Monitoring Committee (JMC), which was set up specifically for monitoring COVID-19 vaccination in real time and on a periodic basis to ensure immediate corrective actions for improvement. 1

Public

www.ifrc.org

Scope of work of Lebanon

➢

Preparatory Phase Whereas each TPM operation will have its own scope and requirements, the key objective of any TPM assignment is to independently monitor the compliance of the vaccination deployment with the NDVP, international standards and donor requirements in terms of supply chain management and administration of COVID-19 vaccines at (i) the key points in the supply chain; and (ii) vaccination sites from the technical, environmental, and social safeguards perspectives.

WHO Guidelines for TPM operation of the COVID-19 vaccination

Wastage: Some wastage is usually expected and accepted as a price to pay for achieving high coverage, especially for vaccines that are presented in multi-dose vials. However, with limited supply, however, every wasted dose represents a missed opportunity for vaccination, and the target for wastage should be close to zero. Additional indicators that may be useful for COVID-19 vaccines are: Vaccines allocated: The number of vaccine doses allocated to a certain region or district as a percentage of the total population, or the percentage of people eligible at a certain phase in the vaccine roll-out, divided by the number of doses in the schedule. This provides a sense of vaccine sufficiency and fair distribution of vaccines in a country.

Understanding the Scope of the TPM Operation

Vaccine availability: Traditionally, programs aim to achieve full vaccine availability and supply to ensure that no vaccination opportunities are missed. In the context of limited supply, stockouts are expected and do not provide a good performance indicator for the supply system.

Vaccines delivered: The number of doses distributed during a certain time period as a percentage of the number of allocated doses for the same period. Administered vaccines: This number can also be compared with delivered vaccines to assess the efficiency of the vaccination process by facility or district. https://www.who.int/

Public

www.ifrc.org

T P M Phase Preparatory Once the scope of work has been agreed upon with the donor and a common clear understanding has been established, the IFRC management team should develop a methodology that is as clear and concise as possible for setting up and implementing the TPM operation. The development of the methodology in close collaboration with the donor agency will foster coordination and collaboration as well as provide an understanding of potential risks and opportunities. The TPM strategy and methodology must be built around key principles that are aligned with the core values of the IFRC. These include particular attention to gender equality, zero tolerance for sexual exploitation and abuse, accountability, cultural sensitivity, and transparency. The three charts below provide further explanations as to vaccine handling and storage, as well as monitoring aspects, health facility-based vaccination, the flow of vaccination at sites and observation, including reporting and recommendations. The assessments include mixed methods of monitoring and data collection.

Vaccination Process

Designing the TPM Methodology

Vaccine Handling and Storage

Health Facility Set-Up

Public

www.ifrc.org

Preparatory Phase A. Vaccine Handling and Storage: This aspect includes recording the vaccine arrival time and monitoring storage points at national and regional levels.

Central

Airport

Warehouse

Observe and Record: ✓ Arrival airport

vaccine

Health Facility

✓ Record time/temperature ✓ Record customs clearance time ✓ Appropriate transportation to the central warehouse (refrigerated trucks and other considerations)

Observe and Record:

✓Arrival time of vaccine at the central warehouse

✓ Daily temperature

Vaccination ✓ Check temperature

✓ Record theVaccination total quantity of vaccine at HF

✓ Move vaccines appropriate freezers ✓Check temperature

freezer

✓ Check entry of vaccine in the

stock register

Observe and Note: ✓ Time dispatch of vaccines from the airport to the central warehouse

✓Daily Temperature Chart (if temperature chart shows major fluctuations in 24 hours, alert your IFRC supervisor) Observe and Report Weekly

Public

✓ Record the time at which the vaccines are taken out from freezers for vaccination ✓ Check stock registers ✓Record time if any vaccine is brought back to freezers at the end of day ✓Record temperature

vaccine

✓ Record temperatures/dispatch time of vaccines daily) to the health facilities (HF)

✓ Check HF freezer Check HF freezer temperature twice daily (before the vaccines are taken out and at the end of the day)

✓Record numbers/temperature of vaccines if returned by HFs

✓ Record and report total daily usage of vaccine.

www.ifrc.org

Preparatory Phase B. Health Facility Set-Up: This aspect shall include three checks: ✓ ✓ ✓

Visibility of vaccination facility Registration Ushering to vaccination booth as per the chart below

Specific checks conducted at the health facility include site requirements as defined by the MoPH, temperature maintenance, stock and inventory management of vaccine and non-vaccine supplies, functionality of equipment and temperature monitoring devices to ensure compliance with cold chain procedures, waste management and related supplies, and security arrangements.

Visibility of Vaccination Facility ✓ Visible banners/signs to guide vaccine recipients to the vaccination facility/area

✓ Staff or volunteers' presence at the entrance to answer questions ✓ Entrance and exit are convenient to pass through ✓ Staff or volunteers are checking that each client is wearing a mask

Registration ✓Number of registration stations set up ✓ Registration stations are Vaccination adequately spaced to ensure social distancing ✓ Staff/volunteers wearing masks

are

Ushering to Vaccination Booth ✓Adequate number of staff or volunteers to usher each recipient to their designated vaccination Vaccination booth ✓ Usherer hands over the registration form of each recipient to the vaccinator

✓ Time it takes for registration for at least 5 recipients daily ✓ A printed form is provided to the usherer for each client to be handed over to the vaccinator.

✓ Entrance and exit have a wheelchair ramp ✓ Staff or volunteers are guiding vaccine recipients to the registration booth.

Public

www.ifrc.org

Preparatory Phase C. Vaccination Process: This aspect is conducted at vaccination sites, and it includes on-site short semistructured interviews for randomly selected vaccine recipients conducted by the TPM field monitors, in-person observation of the vaccination process aided by a checklist/monitoring tool to ensure adherence to vaccination protocols (including eligibility protocols for first and second dose), use of digital registry, eligibility of recipients as per the NDVP, infection prevention on site, AEFI, vaccination recordkeeping and reporting, health worker interaction with beneficiaries, assessment of waste management processes and assessment of security management systems. One field monitor can be assigned for each vaccination site across the project cycle, with each monitor observing the vaccination process at their designated site aided by a mobile application and a checklist.

Vaccination Booth Observe and Record:

Vaccination

Observe and Record:

✓Number of vaccination booths at the HF

✓The total number of vaccinators in the vaccination facility/area

✓Booths are spaced appropriately to ensure social distancing

✓Vaccinators are polite and gentle Vaccination

✓Booths have comfortable chair ✓Adequate ventilation

lighting

and

✓The time between being seated and being vaccinated for at least 5 random clients daily

✓ Vaccinators are appropriate PPE

wearing

✓Vaccinators explain all important information about the vaccine to each client ✓Vaccinators ask critical questions about each client regarding immuno-deficiency disorders and certain allergies ✓Vaccinators provide information about AEFI and when and how to report if needed ✓Presence of an electronic timer for each client and if it is switched on immediately after vaccine is injected (should be switched on for 15 minutes)

Waiting Area/AEFI Observe and Record:

✓ That social distancing is ensured by appropriate spacing of the chairs in the waiting areaIf the Vaccination vaccinators are wearing appropriate PPE ✓ Exit door is clearly visible from the waiting area ✓ Use of the appropriate questionnaire on mobile phones to interview one random vaccine recipient out of every 20 at the exit

✓Vaccine recipient understands that they can leave the waiting area after 15 minutes if they do not have any adverse reaction ✓Presence of staff or volunteers to usher the client to the waiting area after vaccination ✓Each vaccine recipients are handed an interim certificate after the 1st dose of vaccination that must be brought back for the 2nd dose ✓Vaccinator clearly explains the day and time for the 2nd dose to each client

Public

www.ifrc.org

Preparatory Phase Designing the organizational structure of the TPM operation includes determining the human resources needed for a successful operation. While most operations are often limited by resource restrictions, it is important to clearly analyze the scope of the TPM operation to adequately estimate the appropriate budget and required human resources. When developing the organizational structure of the project, it is important to ask two key questions: ✓ ✓

What are the products/outputs required from this project? What are the reporting mechanisms necessary to ensure the development and production of quality products?

Question

Answer

Ensure that the adequate number of field monitors is deployed to each site

Number of sites to be monitored Monitoring frequency

Daily, weekly, monthly

Is the assignment a spot check or daily monitoring?

A. B.

A. Yes B. No A. Paper form B. Digital platform

A. B.

The number of monitors does not need to be the same as the number of monitoring sites The number of monitors must equal to or must surpass the number of sites Hire supervisors Supervisors are not needed Hire data entry clerks Hire an information management officer

Ensure you have data management and analysis

Who will analyze the data?

Ensure the presence of a reporting officer

Frequency and types of reporting

Ensure reports are reviewed before submission One for each function

Who will conduct quality assurance? How many managers are needed? Public

Developing the organizational structure of the TPM operation

How is data collected?

Affects the number monitors required A.

A. Spot Check B. Daily Monitoring

Will monitors be supervised

Implication

www.ifrc.org

Preparatory Phase As with any development or humanitarian activity, each stage of operation could pose different types of risks and involves certain assumptions. It is the duty of the IFRC office responsible for the TPM operation to develop a thorough risk matrix and ensure that it is regularly updated. Key components of the risk matrix could include (but are not limited to): ✓ Purpose: Risk analysis represents an integral part of any emergency response design and implementation, especially in complex environments. Risk analysis and mitigation measures should be planned well ahead by the program team and should be considered throughout the entire project cycle. What are the reporting mechanisms necessary to ensure the development and production of quality products?

✓ How to complete the tool: Identify all possible risks after consulting with local authorities, representatives from Red Cross/Red Crescent branches, community leaders and beneficiaries, local staff and volunteers. Risks are classified under three broad categories: contextual, programmatic an institutional. List as many risks as are relevant to the context. Rank their likelihood form 1 (low) to 5 (high) and then identify the likely impact of each risk, then rank the likelihood from 1-5. Use this

information to feed into TOOL M3_1_4_2 Risk Register Template , including mitigation measures against each risk identified.

Developing and Adjusting the Risk Matrix

Public

www.ifrc.org

Preparatory Phase The Team leader for immediate project is implemented through Field Monitors (FMs). The Field Monitors are responsible for ensuring timely and efficient data collection on all processes of the vaccination roll-out. This includes vaccine delivery, distribution, documentation, and field-level vaccination implementation. FMs are supervised and interlinked by the Field Monitors reporting in case of one or more of the following cases: ✓ ✓ ✓ ✓

Vaccination site IT system crash for more than 30 minutes, Cold chain disruption, Breach of prioritization categories Vaccination not taking place at a site where it should be,

In addition to supervising and supporting the FMs, the Field Monitors Team leader is responsible for the management and support of the field monitors across the different vaccination sites. The Field Monitors Team leader is also responsible for ensuring that field monitors are using an accurate, consistent and reliable recording and reporting system at each vaccination site. The Team leader is also in charge of designing and implementing the deployment of the field monitors to ensure adequate coverage of all stages of the vaccination roll-out process. The FM Team leader participates actively in data verification/cross-checking (data consistency at HF level and field/community level data verification/validation), as per the TORs of TPM and validated monitoring tools.

Designing the Job Descriptions of the Team

Public

www.ifrc.org

Preparatory Phase

Head of the Management Unit

For the Full Jobs Description: Click Here

PMER Assistant

PMER Coordinator Team Leader – Field Monitors Social Specialist

Media

Information Management and Analysis Officer Driver

Human Resources Officer

Administration Officer

Public

www.ifrc.org

Designing the Job Descriptions of the Team

Environmental and Social Specialist

Implementation Phase TPM operations are a new activity to the functions and focus of the IFRC. Nonetheless, the IFRC is a long-standing humanitarian organization with expertise and local knowledge in all the communities where it operates.

✓ Technical and operational proactivity. IFRC teams responsible for the TPM should act proactively in the design of the TPM operation and data collection tools. This could be done by soliciting support from technical experts at the regional and HQ levels. ✓ IFRC should develop strong partnership modalities with donor agencies that include regular formal and informal communication mechanisms. ✓ As a trusted impartial actor, the IFRC should maintain this position while striking a delicate balance to ensure visibility and recognition of its role and activities.

However, not all TPM projects start within a state of emergency, and, therefore, some activities can be done in parallel for efficient use of time and resources.

Below are two examples of how country or regional offices could plan implementation processes to ensure smooth operation.

Public

www.ifrc.org

Implementation Phases and Sequencing

TPM projects often begin in an emergency setting, requiring quick action and parallel functioning of the different components. Field monitors and managers are often required to hit the ground running. In the case of Lebanon, the contract was signed two days before the beginning of operations, requiring the IFRC to jump start the project through existing staff members in the regional office while the TPM team/staff was being recruited.

Implementation Phase Emergency TPM Operation Steps Activity

Responsibility Country office or Regional Office Management

Step 2 (a): Develop or adapt data collection tools to the required context Develop or adapt reporting formats based on donor requirements Start exploring digital platforms (as soon as possible)

IFRC PMER in collaboration with Donor agency

Step 2 (b): In parallel, develop a project organizational structure, finalize job descriptions and start the recruitment of TPM teams

IFRC Resources

Step 3: Field deployment for data collection

Country office or Regional Office Teams (or both)

Step 4: Train newly recruited TPM team (Field monitors and supervisors)

IFRC Human Resources IFRC Technical Team

Step 5: Hand over operation to TPM Team

IFRC country office or regional office and TPM team

Step 6: Monitor the operation

IFRC Country team or regional team

Public

www.ifrc.org

Human

Implementation Phases and Sequencing

Step 1: Negotiate and Sign Contracts with Donor

Implementation Phase

✓ ✓ ✓ ✓ ✓

Public

Scale of the operation (10 sites vs 40 sites) The level and type of support that can be accessed by IFRC HQ/IFRC Regional offices Availability of subject matter expertise Availability of resources (data collection tools, formats from other countries, etc.) Ability to provide support quickly (country access by international staff, mobility, security situation in country of operation)

www.ifrc.org

Implementation Phases and Sequencing

The above steps are only an illustration of how the operation is likely to progress in different contexts and environments. Some key points and questions to consider when developing the implementation sequence of the TPM operation include:

Implementation Phase Lessons Learned from Recruitment in Lebanon’s TPM Ensure the adequate number of Field Monitors and Field Supervisors Ensure that Field Monitors and Supervisors have the necessary

Key Considerations Recruitment

the

technical background Recruit the entire team at once

It is important to ensure adequate recruitment of personnel at the start of the operation. The absence of the adequate number of field monitors would result in a heavy workload on some staff, potentially leading to burnout and exhaustion. However, the alignment of the recruitment process to project needs will support the comprehensive design of training programs, while ensuring that all field monitors clearly understand the tools and processes.

Make sure that each member is aware

their

roles

responsibilities Shar e

the

and

w or kflow

info r m atio n

m anage me nt w ith m em ber s

syste m

all –

team so

that

e ach wo uld kno w how a

de lay

affe cts

t he

e ntir e o pe r atio n

Depending on the nature of the TPM, it is imperative that the field monitors have some background knowledge of the sector they will be observing, as well as relevant knowledge of the tools that will be used (a sample JDs of the different functions of the COVID-19 vaccine in Lebanon is available in Annex 1).

Public

www.ifrc.org

and

the

TPM

organigram and ensure clear reporting lines

Recruitment and Training of Field Monitors and Field Supervisors

Although retention cannot be guaranteed for staff, quality and structured documentation could ensure that institutional memory is intact and that work progresses smoothly, therefore promoting continuity.

if possible

Implementation Phase In addition to adequate planning, the training of field monitors is the second most important step in setting up a successful TPM operation. Unlike traditional IFRC deployments, the quality of the TPM products is directly determined by the quality of data collected by the field monitors. Hence, the training of field monitors is critical for the successful implementation of the TPM operation. In addition to the technical training of the field monitors on the tools of data collection, there are key trainings that should also be implemented to ensure that field monitors are aware of their role and act in a way that is compatible with the code of conduct and spirit of the IFRC. The trainings for field monitors should include: 1. IFRC Code of Conduct 2. IFRC anti-fraud policy 3. Preventing Exploitation and Sexual Abuse (Annex 2 includes a sample presentation on PSEA training for field monitors in Lebanon) https://www.who.int/publications/i/ item/WHO-2019-nCoV-Vaccine_ deployment-2020.1

Lessons Learned from Field Monitors Training in Lebanon’s TPM Technical Training on data collection tools needs to be delivered as early as possible in the TPM process

●

Field Monitors should not be allowed in the field without comprehensive training on and testing of data collection tools

●

PSEA training is an important step to reduce any risks to IFRC during the work of field monitors

●

Technical training requires coaching and mentoring. FMs technical training should include:

✓ 2-day training on data collection tools ✓ 2 days shadowing data collection done by experienced staff

Training of Field Monitors

●

✓ 1 day implementing the tools (i.e. collecting data) while being supervised by experienced staff ✓ FMs cleared to start data collection

Public

www.ifrc.org

Implementation Phase

How to ensure the efficiency and safety of field monitors on the ground – Lessons Learned from Lebanon

✓ Ensure that field monitors are very well trained and that there is a common understanding of the data collection tools. This could be done by sending two field monitors as part of the training to the same site and comparing the data collected by each and the errors corrected. ✓ The field monitors training should include a strong component on the difference between “monitoring” – i.e. the job requirement – and the traditional role of IFRC as a first respondent in emergencies.

Training of Field Monitors

✓ Ensure that field monitors are wearing IFRC vests, with an “Observers” badge as some communities may perceive them as aid providers or rescue workers and may appear confused by their lack of action.

✓ The training should also include aspects of staff self-care, access to psychosocial support and other staff wellbeing schemes.

Public

www.ifrc.org

Implementation Phase The design and the number of data collection tools will be determined based on the type of monitoring required (daily, spot check); the sector (health, hygiene); and the overall requirements of the donor.

The TPM operation in Lebanon focused on the following aspects:

Area / Site of assessment

Methodology Spot-check: In-person observation, aided by a checklist/monitoring tool

Vaccination sites – equipment and supplies

On-site independent observer: In-person- observation aided by a checklist/monitoring tool

Vaccination sites – service delivery

On-site independent observer: In-person observation of vaccination process aided by a checklist/monitoring tool

Service provider and client perspectives and feedback

On-site independent observer conducting short, semi-structured interviews for randomly selected vaccine recipients

Social media and GRM monitoring

Staff of TPM monitor social media and analyze the data from the call center set up by MoPH

Implementation of ESMF

Staff of TPM will monitor implementation of E&S requirements as per the E&S disclosed instruments

Stock monitoring

Daily stock count and reporting, as well as comparison with the data as per the information system and data issued through the digital monitoring systems

Public

Data Collection Tools

Vaccine arrival and vaccination storage points at national and regional/government levels

www.ifrc.org

Implementation Phase

For COVID-19 vaccination monitoring in Lebanon, there were five main tools for data collection (the data collection tools used in Lebanon TPM operation are presented in Annex 3) as follows:

Data is collected on a daily basis from the vaccination sites and entered in real time on KoBo Toolbox, which is linked to the Power Bi Dashboard. Below is an overview of the data management system developed by the IFRC to ensure data quality and credibility.

Public

www.ifrc.org

Data Collection Tools

A. COVID-19 Vaccination Site Checklist B. COVID-19 Vaccine Recipient Interview C. Health Provider/Vaccinator Interview D. COVID-19 Vaccination Environmental and Social Safeguards Checklist E. COVID-19 Vaccine Arrival and Central Warehouse Checklist - Vaccine Arrival (on every scheduled arrival)

Implementation Phase Data Collection Using Kobo Toolbox Forms ✓ Field monitors use the Kobo Data Collection Tools to collect data from each vaccination site visited on a daily basis. ✓ All the checklists/questionnaires are uploaded on the IFRC server on Kobo Toolbox. ✓ Each field monitor has their own credentials to access and submit Kobo forms.

Data Management, Cleaning, and Export ✓ Data cleaning is done directly on the Kobo database, since the Power BI Dashboard relies on live data.

✓ Data for each questionnaire is exported in a separate Excel file for the analysis to begin.

Data Analysis

Data Collection Tools

✓ A deadline is set for submission by field monitors which allows for sufficient time for data processing and report development.

✓ All the indicators are analyzed into Pivot tables, with the list of the vaccination sites on rows, and the dates and indicators in columns. ✓ For every indicator in the different checklists, there are 2 Pivot tables – one in numbers and the other in percentages.

Public

www.ifrc.org

Implementation Phase Data Visualization and IM Products ✓ When the data is ready, users can select from the different checklists the indicators needed to develop analytical products. ✓ All the needed indicators are listed in tables in a Master Analysis Sheet weekly. ✓ All the visuals needed for the narrative reports are created and formatted as needed in the same sheet. ✓ This data is also used to develop the infographic cards that are posted on social media, mainly on Twitter.

Social media

For manual monitoring, Facebook, Instagram, YouTube and Google search queries are monitored using keywords, hashtags, location sites, media outlets, and replies on official pages. Relevant data is captured, coded, and categorized for sentiment and topics on an Excel file that is used for the analysis of the content. ✓ Tools/potential tools: Brandwatch/Talkwalker/Tweetdeck ✓ Monitoring social media channels (Twitter, Facebook, Instagram, YouTube) in English & Arabic: o Official channels on different platforms (IFRC/World Bank/MoPH) + replieso on them o Influencer lists o Hospital location pages o Keywords o Hashtags o Media outlets on different platforms

Data Collection Tools

Social media and media monitoring is done using both automated and manual search. The automation process relies on listening tools (potential tools: Brandwatch/Talkwalker) by using optimized queries, generating results mainly from Twitter. Tweetdeck is a monitored software that is also used which eases the input of Twitter keywords, hashtags and replies on official pages.

✓ Google News for articles

Public

www.ifrc.org

Implementation Phase Lessons Learned from Data Visualization in Lebanon’s TPM ●

It is important to ensure that all information and templates are developed before data visualization starts.

●

Explore possible data visualization systems that could help the process, such as maps and charts, or use https://www.arcgis.com/index.html which enables more functions.

●

Ensure that the checklists are designed in a way that would enable reporting as well as data visualization.

●

Clarify roles, reporting lines, and data flow to ensure that data is received by the individuals responsible for data visualization accurately and quickly.

Data Collection Tools

www.ifrc.org

Public

Reporting Phase During the planning phase, the IFRC will agree with the donor on the following key issues:

Frequency reporting

Identification and definition of issues that require an “incident” report

Reporting formats (taking into account that the format would influence the data collection tools)

Reporting process

The below reporting processes were adopted by the IFRC regional office in Lebanon in consultation with the donor (World Bank). Each TPM operation should develop its own reporting processes to ensure relevance to the context.

✓ IT system crash (not available for >30 minutes) ✓ Cold chain disruption (throughout the supply chain): At airport, transport from airport to the central UCC, central UCC, transportation of vaccines, storage at the vaccination sites.

Reporting Flow

The following incidents were identified as “urgent” and requiring immediate reporting by IFRC and WB in Lebanon:

✓ Repetitive or significant breach of prioritization categories, including misuse of mobile clinics

✓ Major incident disrupting vaccination, including long power cut, major security incident, etc. ✓ Vaccination not taking place at all at a site where it should ✓ Vaccination taking place at a site where it should not (not included in list of vaccination sites for the week)

Public

www.ifrc.org

Reporting Phase Daily Reporting Flow:

IFRC shares the dailyshares report with IFRC the daily report with WB (timing WB (timing to be to be agreed between agreed IFRC & donor) between IFRC & donor)

Reporting Flow

Donor shares questions, modifications with IFRC (timing to be agreed between IFRC & donor)

IFRC shares revised report with MoPH and published the revised report (depending on agreement between IFRC, donor, and MoPH)

Public

www.ifrc.org

Reporting Phase Weekly Reporting Flow:

IFRC shares with WB: ✓ 1-weekly narrative report (covering Mon-Sat) ✓ 2-PPT presentation summarizing weekly report ✓ 3-Infographic <every Monday before 1pm>

WB shares questions, modifications with IFRC <every Monday before 6pm>

modifications with IFRC <every Monday before 6pm>

Reporting Flow

WB shares questions,

WB shares revised weekly narrative report with JMC, National Vaccine Committee IFRC shares revised weekly narrative report with MoPH focal point <every Tuesday before 2pm> IFRC presents summary of findings using revised PPT presentation in JMC meeting and National Vaccine Committee meeting <every week> IFRC makes revised infographic public <every Tuesday before 6pm>

Public

www.ifrc.org

Reporting Phase Incident report process

1.In the case of urgent** issues/ incidents picked up by IFRC:

1: IFRC shares reported incident with WB and MoPH focal point by phone 2: An incident report is shared with WB <end of the day>

4: IFRC shares revised incident report with MoPH focal point

Public

www.ifrc.org

Reporting Flow

3: WB shares questions, modifications with IFRC <next day morning>

Reporting Phase Incident report process

2.In the case of urgent** issues/ incidents picked up by WB:

Reporting Flow

Public

www.ifrc.org

Reporting Phase Reporting Formats

Depending on the agreement with the donor, reporting could take place daily, weekly, and/or monthly. The development of reporting forms and templates would ensure that the required data is being systematically collected and that data analysis approaches support the production of reports, reduce time and effort and promote adequate communication between IFRC and the donor. A COVID-19 vaccination dashboard could be developed to provide insights into a variety of programmatic aspects and vaccination data, and to serve as a useful communication tool. For example, the dashboard could show key performance indicators, including data on service availability and readiness (human resource capacity, cold chain and supply), vaccine uptake and coverage, AEFI, etc. The vaccination component could also be part of a broader COVID-19 dashboard that would include surveillance (cases and deaths). Designing a dashboard and considering what information to include in it is also a useful exercise to help determine what data needs to be collected.

Daily Report Template

Part 1. Monitoring Performance Number of vaccination sites monitored, percentage of sites visited out of the active sites, frequency/dates of visits to the central warehouse – 1st source KOBO – verification Excel sheet

Number of monitors deployed - – 1st source KOBO – verification Excel sheet Number of health providers interviewed – KOBO + GIS (if used) Number of vaccine recipients interviewed – KOBO + GIS (if used)

Part 2. Key Observations Enabling factors – 1st source feedback from FMs Challenges encountered – 1st source feedback from FMs Any incidents that might have occurred during the day – feedback from FMs

Public

www.ifrc.org

Reporting Phase Weekly Report Template

Part 1. Monitoring Performance Possible Graphs: Percentage of Vaccination sites monitored per day – Total Field Monitors deployed per day

Key Findings by Thematic Area 1. Vaccine arrival

2. Vaccine storage at central warehouse and transportation

Observations and Findings 1. Vaccination sites – Vaccine storage, handling, and waste management 2. Vaccination sites – Vaccine recipient journey 3. Healthcare provider perspectives 4. Vaccine recipient perspectives 5. Grievance monitoring 6. Media monitoring 7. Environmental and social safeguards monitoring

Public

www.ifrc.org

Reporting Phase

➢ ➢ ➢ ➢ ➢ ➢ ➢

In developing the TORs for the evaluation of the TPM project, it is important to determine: The type of evaluation (summative, formative, process, performance) The purpose, objective, and scope of the evaluation The required deliverables (inception report, summary, final report, etc.) The users of the evaluation Evaluation criteria and questions Expected evaluation methodology and approach

Evaluation Phase

In developing the TORs for the evaluation of the TPM project it is important to determine

Annex 4 includes a sample TOR for a TPM operation.

Public

www.ifrc.org

Reporting Phase

Lessons Learned and Best Practices from the Lebanon TPM

➢ ➢ ➢

➢ ➢

➢

Public

Recruitment and training of field monitors is imperative for the successful implementation of the TPM mission. The technical background of field monitors is key for success. Building internal expertise and strength in PMER is a priority for the activities of the TPM Align expectations with abilities, i.e. ensure a common understanding between donor expectations and availability of resources in country office or regional office or both. Plan, Plan, Plan and then Implement. TPM skills and expertise need to be part of an existing training curricula that prequalifies some volunteers, delegates, and other staff on the required skillsets for TPM management. TPM skills can be part of a curriculum that exists within the Federation, perhaps as part of the community engagement sector and how to work with local communities.

www.ifrc.org

Lessons Learned

➢