CHISHPIN PROJECT FINAL EVALUATION by United Purpose

6/25/2019

CHISHPIN PROJECT FINAL EVALUATION EVALUATION REPORT

Mathew Ocholi and Johnson Eteng CONSULTANTS

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

Table of Contents Executive Summary ............................................................................................................................................. 6 1.0 Introduction. .................................................................................................................................................. 9 1.1 Purpose of the Evaluation ......................................................................................................................... 9 1.2 Evaluation Objectives ................................................................................................................................ 9 1.3 Organisation context ................................................................................................................................. 9 1.4 Logic and Assumptions of the Evaluation ............................................................................................... 10 1.5 Overview of UK Aid Direct funded activities .......................................................................................... 11 2.0 Evaluation Methodology ............................................................................................................................. 11 2.1 Evaluation plan ........................................................................................................................................ 12 2.1.1 Sample Size ....................................................................................................................................... 14 2.2 Strengths and weaknesses of selected design and research methods .................................................. 15 2.3 Summary of problems and issues encountered ..................................................................................... 15 3.0 Findings ........................................................................................................................................................ 16 3.1 Overall Results ......................................................................................................................................... 16 3.2 Assessment of accuracy of reported results. .......................................................................................... 31 3.3 Relevance ................................................................................................................................................. 32 3.4 Effectiveness ............................................................................................................................................ 35 3.5 Efficiency .................................................................................................................................................. 38 3.6 Sustainability............................................................................................................................................ 38 3.7 Impact....................................................................................................................................................... 39 4.0 Conclusions .................................................................................................................................................. 40 4.1 Summary of achievements against evaluation questions ...................................................................... 40 4.2 Summary of achievements against rationale for UK Aid Direct funding ............................................... 41 4.3 Overall impact and value for money of UK Aid Direct funded activities ............................................... 42 5.0 Lessons learnt (where relevant) .................................................................................................................. 42 5.1 Project level - management, design, implementation ........................................................................... 42 6.0 Recommendations ....................................................................................................................................... 43

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

LIST OF ABREVIATIONS

CHISHPIN

Community-Led Health Improvement through Sanitation and Hygiene Promotion in Nigeria

United Purpose

LGA

Local Government Area

ToR

Terms of Reference

SDG

Sustainable Development Goals

ODF

Open Defecation Free

DFID

Department for International Development

CLTS

Community Led Total Sanitation

Impact Evaluation

CSO

Civil Society Organization

Concern Universal

WASH

Water, Sanitation and Hygiene

RUSHPIN

Rural Sanitation and Hygiene Promotion in Nigeria

Household

Country Director

NTGS

National Task Group on Sanitation

MDG

Millennium Development Goals

FGD

Focus Group Discussion

DGSA

Director General Services and Administration

HOLGA

Head of Local Government Administration

KII

Key Informant Interview

WASHCOM

Water, Sanitation and Hygiene Committee

RUWASSA

Rural Water Supply and Sanitation Agency

EHC

Environmental Health Club

NGO

Non-Governmental Organization

ORT

Oral Rehydration Therapy 2

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

CRS

Cross River State

CMT

Country Management Team

INGO

International Non-Governmental Organization

United Kingdom

VFM

Value for Money

Natural Leader

PAC

Project Advisory Committee

AFRUD

Action for Rural Development

LEF

Life Empowerment Foundation

NYSC

National Youth Service Corps

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

Table of Figures Figure 1: Number of Households reached per project LGA ............................................................... 17 Figure 2: Occupational distribution among the respondents ............................................................ 17 Figure 3:Types of Latrines in Sampled Households............................................................................ 18 Figure 4 Community Source of Water ................................................................................................ 18 Figure 5 Percentage of Latrines with Drop Hole Cover ...................................................................... 19 Figure 6 Latrines observed with Faeces on the Floor ........................................................................ 19 Figure 7 Types of Latrine Superstructure Materials ........................................................................... 19 Figure 8 Number of Handwashing Materials Observed ..................................................................... 20 Figure 9 Presence of Soap or Ash by Handwashing Station .............................................................. 20 Figure 10 Use of Handwashing Facilities ............................................................................................ 21 Figure 11 Presence of Faecal Matter around the Compound ............................................................ 21 Figure 12 Schools with Latrines and those without ........................................................................... 21 Figure 13 Schools with and without EHCs .......................................................................................... 22 Figure 14 Schools that Participated the Global Handwashing Day events organised by UP ........... 22 Figure 15 Under-five Diarrheal cases reported in Wanikade Health Centre in Yala LGA ................. 23 Figure 16 Under-five Diarrheal cases reported in Nde Health Centre in Ikom LGA ......................... 24 Figure 17 Biase LGA Under-five Diarrheal cases reported to State Epidemiology Unit ................... 24 Figure 18 Ikom LGA Under-five Diarrheal cases reported to State Epidemiology Unit .................... 25 Figure 19 Yala LGA Under-five Diarrheal cases reported to the State Epidemiology Unit ............... 25 Figure 20 Overall LGA Under-five Diarrheal cases reported to State Epidemiology Unit ................ 26 Figure 21 Gender Distribution in WASHCOM Executives in the Project LGAs .................................. 35

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

List of Tables Table 1:Proposed Sample Communities distributed by ward and LGAs ........................................... 14 Table 2 Responses from School Children on Benefit of the GHWD celebrations ............................. 23 Table 3: Showing the Population Verification of Biase LGA ............................................................. 27 Table 4: Showing the Population Verification of Ikom LGA ............................................................. 28 Table 5: Showing the Population Verification for Yala LGA .............................................................. 29 Table 6 Summary of Population Verification for the three LGAs ...................................................... 30 Table 7: Extracts of FGD health response from the women group ................................................... 33 Table 8:Extracts of FGD health response from men group ................................................................ 33 Table 9:Extract of FGD health response from School Children ......................................................... 33 Table 10:Extract of KII health response from Traditional leaders ..................................................... 33 Table 11:Extracts of KII health response from School Teachers........................................................ 34 Table 12:Women involvement in the implementation of CHISHPIN project ................................... 34

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

Executive Summary United Purpose (UP), formerly Concern Universal, has been implementing the Community-Led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Project since November 2015. The project rounded up on 31st March 2019. As part of the project closure process, UP commissioned a Final Evaluation of the CHISHPIN Project. The purpose of the independent final evaluation for the CHISHPIN project is to generate information from analysed data collected from the beneficial communities, implementation partners, Local Government Area (LGA), State level, National partners and sector actors and produce a report that answers all the elements of the Terms of Reference (ToR); provides findings, conclusions and recommendations that are based on robust and transparent evidence for the understanding of the project for the Fund Manager. The evaluation set out to achieve three explicit objectives: To independently verify (and supplement where necessary), United Purpose’s record of achievement as reported through its Annual Reports and defined in the project log frame which includes;

•

Verify target population through selective sampling in the three Local Government Areas (LGAs).

2. To assess the extent to which the project was good value for money, which includes considering: •

How well the project met its objectives;

•

How well the project applied value for money principles of effectiveness, economy, efficiency in relation to delivery of its outcome;

•

What has happened because of Department for International Development (DFID) funding that wouldn’t have otherwise happened; and

•

How well the project aligned with DFID’s goals of supporting the delivery of the Sustainable Development Goals (SDGs).

3. Identify unique characteristics for United Purpose performance in the delivery of Local Government Area (LGA) wide Open Defecation Free (ODF) as compared to other Community Led Total Sanitation (CLTS) practitioners in Nigeria. If there are any substantial differences, (possible explanations should be provided). CHISHPIN project learnt from the RUSHPIN programme and adapted the strategies and innovations to achieve two (2) full LGAs Open Defecation Free (ODF) with the third more than 50% in its three (3) years of implementation. The CHISHPIN project was implemented LGA-wide with the aim of reducing under-five mortality by reducing diarrhoea prevalence through improving sanitation coverage and hygiene practices using the Community-led Total Sanitation (CLTS) approach. The methodology adopted for this Impact Evaluation (IE) is the mixed method approach using Quantitative and Qualitative methods with strong emphasis on triangulation. A contribution-based approach was adopted for this Impact Evaluation (IE). A comprehensive package containing possible contributory factors to the expected outcomes from the CHISHPIN project was identified. 34 communities were sampled in Biase, 39 in Ikom and 45 in Yala LGAs. In total 3,869 households were proposed to be sampled, but eventually 2,541 were sampled.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report 76.5% of the respondents in the households visited are Farmers, Civil Servants are 10.2% percent, while those in business are 4.4%. Number of persons in the compounds reached is 18,741, 8,269 are men, 10,472 are women and 3,109 are children. 2,698 latrines were found in the 2,541 HHs reached. A population verification exercise was carried out in 108 communities in line with the objective of the evaluation. While there were variations between some of the populations documented by UP and those at the community level this was not wide spread. Overall, the variance between previous population data of the project communities collected by UP and that collected by the evaluation directly from the communities is 1,728 people which represents about 6% of the UP figures. Of the 2,698 toilet facilities identified in the compounds, 2,484 are in use showing 92% usage rate. Two types of latrines were predominant in the sampled households. 70% are pit latrines while 26% are Water Closet system. Water supply seems to be a challenge in the communities. 57.4% of the respondents say they walk more than 20 minutes to get water while 35.2% say they walk less than 20 minutes to fetch water. Poor access to water can constitute a demotivation for community members to WASH their hands at critical times.

UNDER-FIVE DIARRHEAL CASES IN WANIKADE YALA LGA Response

50 33 17

2015

2016

2017

2018

Years

Under-Five diarrhoea reported cases based on data obtained from two health centres in Yala LGA.

Under-Five Diarrheal Cases in Nde Health Centre Ikom LGA Number of Cases

52 28 13

2015

7 2016

2017

2018

Years Under-Five diarrhoea reported cases based on data obtained from two health centres in Ikom LGA

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Health records for under-five children could only be found in two Health Centres (one in Yala and the other in Ikom) out of the 18 visited. The trend between 2015 and 2018 in these two results clearly shows a steady decline in diarrhoea cases reported. Monthly health data, between 2016 and 2018, reflecting diarrheal in under-five children, obtained from the State Epidemiology Unit in Calabar, shows a close relationship with the trend obtained with the data from two health centres. While these show general indications of a downward trend, they are not enough evidence upon which to base contributory causal inference, it is however an indication that such a pattern of downward trend may exist in the communities where the project was implemented. Testimonies from the community beneficiaries is however overwhelming. They testify about ,no more diarrhoea incidences for their children, reduction in health expenditures, no more cholera in the communities since the CHISHPIN project started. It is clear from this evaluation that the CHISHPIN project is a success in that it has achieved the objectives it set out to achieve within the three years of its implementation. It is very relevant, effective, efficient and has made visible impact on the lives of people. However, in the long run, the project, communities will face the challenge of sustaining the gains recorded during the project lifespan, this is because money to provide continuous support to the community (through regular visits by LGA WASH departments or CSOs) is not available outside UPs financing and there is no indication money will be available in the nearest future. It is not therefore clear if the communities will continue to sustain the changed behaviour if they are not followed up. There is need to engage CSOs and CBOs with capacity to provide needed support in addition to the LGA WASH units The project demonstrated good value for money by recognizing the key cost drivers in the project, doing things the right way and demonstrating cost-effectiveness and focusing on them to ensure cost savings in the implementation of the project without compromising on the quality of service. It is recommended that adequate funding be provided for projects like the CHISHPIN project to disseminate widely its successful approaches.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

1.0 Introduction. United Purpose (UP), formerly Concern Universal, has been implementing the Community-Led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Project since November 2015. The project rounded up on 31st March 2019. As part of the project closure process, UP commissioned a Final Evaluation of the CHISHPIN Project. This report is the output of the evaluation carried out on the project. It covered the three (3) project Local Government Areas (LGAs) of Biase, Ikom and Yala in Cross River State.

1.1 Purpose of the Evaluation The independent final evaluation reports for the CHISHPIN project will be used to inform the Fund Manager’s understanding of the United Purpose’s performance at the project level and will also be used to inform the Evaluation Manager’s assessment of performance at the UK Aid Direct fund level. It answers all the elements of the Terms of Reference (ToR); provides findings, conclusions and recommendations that are based on robust and transparent evidence; and where necessary supplements United Purpose’s own data with independent research.

1.2 Evaluation Objectives The evaluation set out to achieve the following objectives: To independently verify (and supplement where necessary), United Purpose’s record of achievement as reported through its Annual Reports and defined in the project log frame which includes;

•

Verify target population through selective sampling in the three LGAs.

2. To assess the extent to which the project was good value for money, which includes considering: •

How well the project met its objectives;

•

How well the project applied value for money principles of effectiveness, economy, efficiency in relation to delivery of its outcome;

•

What has happened because of DFID funding that wouldn’t have otherwise happened; and

•

How well the project aligned with DFID’s goals of supporting the delivery of the SDGs.

3. Identify unique characteristics for United Purpose performance in the delivery of LGA wide ODF as compared to other CLTS practitioners in Nigeria. If there are any substantial differences, (possible explanations should be provided).

1.3 Organisation context United Purpose (UP), former Concern Universal (CU) is an international development charity with an innovative community-led approach to delivering the sustainable development goals (SDG) and eradicating global poverty and inequality. UP works at grassroot levels with communities to lift them up out of poverty by providing solutions that last. With its unique approach that combines high quality intelligent development, disruptive innovation and enabling independence, UP pushes boundaries of innovation and continuously improve its 9

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report community driven approach, while taking it to scale. UP’s WASH programmes identify best practices, pilot innovation and scale up successful methodologies. UP’s flexibility enables it to adapt different approaches to the context in which it does it’s work in Nigeria, allowing the organization to achieve sustainable results UP promotes inclusive service provision and interventions that ensure the participation of vulnerable and disadvantaged groups, believing that behaviour change is key to sustained improve health. UP has demonstrated its WASH capabilities in the sector in Nigeria. Recently, UP implemented the Rural Sanitation and Hygiene Promotion in Nigeria (RUSHPIN) programme in Benue and Cross River States resulting in the first ever declared Open Defecation Free (ODF) Local Government Area (LGA) in Nigeria. In 2015, UP began the replication of the success story in Obanliku LGA in three LGAs of Biase, Ikom and Yala with funding obtained from the Department for International Development (DFID) under the UK Aid Direct Impact window. UP was the Grant Holder with co-funding expected from the Cross River State Government and other sources. As has been the case within the Nigeria sector, the Government co-funding was never realised as a result of lack of commitment from the Government of Cross River State. CHISHPIN project learnt from the RUSHPIN programme and adapted the strategies and innovations to achieve two (2) full LGAs ODF with the third more than 50% in its three (3) years of implementation. The CHISHPIN project was implemented LGA-wide with the aim of reducing under-five mortality by reducing diarrhoea prevalence through improving sanitation coverage and hygiene practices using the Community-led Total Sanitation (CLTS) approach for the entire population of three LGAs of Biase, Ikom and Yala in Cross River State, Nigeria. The project also aimed to strengthen institutional capacity for sustainable sanitation and hygiene service delivery at the community, local, and State government level as well as increasing knowledge and awareness of improved hygiene practices in target LGAs.

1.4 Logic and Assumptions of the Evaluation CHISHPIN project does not have a theory of change. The logic of the project is guided by a Logical Framework (logframe). The Logframe clearly identified inputs, outputs and outcome that the project was designed to achieve. The logframe also clearly specified milestones and targets with clear timelines for delivery. This evaluation focused on understanding the following; •

The connections between the inputs, activities and outputs

•

The planned impacts and outcome objectives

•

The appropriateness of the logic behind the CHISHPIN project intervention which in this case is achieving health impact through eradication of open defecation in an LGA.

•

The evaluation also examined the underlying assumptions made in the logframe in order to understand the appropriateness within the context of the project.

•

The evaluation also examined if there are any unintended or unexpected outcomes or impacts as a result of the CHISHPIN project in the three LGAs.

This evaluation is guided by the funders standard definition of Impact Evaluation (IE) as outlined in the Stern et al. (2012) working paper 38. In this paper DFID defined IE as follows; ▪

evaluating the positive and negative, primary and secondary long-term effects on final beneficiaries that result from a development intervention;

▪

assessing the direct and indirect causal contribution claims of these interventions to such effects especially for the poor whether intended or unintended; and 10

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report ▪

explaining how policy interventions contribute to an effect so that lessons can be learnt.

1.5 Overview of UK Aid Direct funded activities UK Aid Direct was established in 2014 as a successor to the Global Poverty Action (GPAF) which was created in 2010. It is funded by the UK‘s Department for International Development (DFID). UK Aid Direct is a challenge fund designed to support the UK’s commitment to achieving the Global Goals. As at February 2018, UK Aid Direct has reached more than 3 million people through 147 grants across 31 countries. The aim of UK Aid Direct is to fund small and medium sized national and international civil society organizations (CSO) to reduce poverty and works towards achieving the Global Goals. UK Aid Direct funding reaches the most marginalized and vulnerable population supporting DFID’s agenda to ‘leave no one behind.’ UK Aid’s agenda can be achieved through funding projects that encompass service delivery, economic empowerment, strengthening accountability or generating social change.

2.0 Evaluation Methodology A contribution-based approach was adopted for this Impact Evaluation (IE). A comprehensive package containing possible contributory factors to the expected outcomes from the CHISHPIN project was identified. This recognised other possible interventions that may contribute to the reduction in the prevalence of diarrhoea in the target LGAs. The methodology adopted for this IE is the mixed method approach using Quantitative and Qualitative methods with strong emphasis on triangulation. Structured questionnaire was used to collect quantitative data while key informant interviews, focus group discussions, structured observations were used to collect qualitative data. Also, the evaluation process identified the various outputs, outcomes and milestones from the logframe and verified whether the activities impacted on the trends of problems and made a difference, and how much it addressed the goal after the project lifecycle. The evaluation process determined adherence to the sustainability guidelines covered in the proposal and logframe. The evaluation sought answers to the questions outlined in the evaluation ToR using the information gathering tools mentioned above. Mixed method approach adopted in this evaluation involved combining qualitative and quantitative data collection and analysis approaches with the evaluation process. By adopting a mixed method approach in this evaluation, the consultant combined the internal validity of quantitative methods and external validity of qualitative research to obtain a productive result. By mixing the methods it was possible to arrive at a richer and more complete description of phenomenon than by using a single approach. Multiple methods helped reconstruct baseline data and overcome time and budget constraints. Mixed method approaches combined quantitative approaches that permit estimates of magnitude and distribution of effects, generalization and tests of statistical differences with qualitative approaches that permit in-depth description, analysis of processes and patterns of social interaction. These integrated approaches provided the flexibility to fill in gaps in the available information, to use triangulation to strengthen the validity of estimates, and to provide different perspectives on complex, multi-dimensional phenomena (Bamberger, Rao and Woolcock, 2010).

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

2.1 Evaluation plan The CHISHPIN end of project evaluation was commissioned by United Purpose, the grant holder of the UK Aid Direct funding to fulfil the project closing requirements. CHISHPIN is a project that set out to ultimately reduce diarrhoea in under-five, through the CLTS process that facilitates poor and marginalized communities to construct latrines and imbibe effective handwashing as a result of a sustained behaviour change. The purpose of the evaluation is to determine if the objectives set out to be achieved by the CHISHPIN project was achieved. It did this by collecting information that helped to address the evaluation questions set out in ToR. Time is a constraint in the evaluation of this nature. There is also the challenge of obtaining the required data to establish causation. That is establishing that the project intervention is solely responsible for the effect that is claimed. This evaluation involved a selection of beneficiary communities in the three project LGAs, other project stakeholders at LGA, State and National levels within the WASH sector. This evaluation did not extend to communities outside the project areas because it is not designed as an experimental or quasi-experimental process. In order to ensure a comprehensive evaluation of the CHISHPIN project the plan examined the objectives and the questions to determine the tools/methods to adopt for collecting the required information. The first task was to independently verify (and supplement where necessary), United Purpose’s record of achievement as reported through its Annual Reports and defined in the project log frame which includes; •

To verify target population through selective sampling in the three LGAs.

The methods used in evaluating the project included the following: •

Desk review of documents: UP’s annual reports and progress against logframe were reviewed, M&E reports, including the data collected and imputed to produce the reports were also reviewed to determine progress made through the course of the project implementation and ascertain the quality of documented results.

•

To verify target population in project communities, three (3) communities per ward in each LGA were targeted, making a total of 108 in the three LGAs. A simple questionnaire with limited questions to elicit the information from households was used. Surveyors administering Household (HH) surveys were also responsible for verification of the population in each compound/HH visited in the target communities and obtained information about the number of people in the household or compound segregated by gender and age groups. Even though the questionnaire was used for eliciting information on the number of people in a household, the challenge of lack of availability of households affected the reliability of information obtained. However, because community WASHCOM also kept a register of households commitment list which provided an up to date information on the population in each household in the community, this was used for the verification exercise. The population figures collected by the community in its household commitment register was used to verify the population data obtained by UP during project implementation through its peer review process and the result is presented in the findings section. The essence of this verification is to determine the reliability of the population information used by UP in the project.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report •

Field data collection to determine the impact of the project on the beneficiaries, implementers of the project and the sector.

To assess the extent to which the project was good value for money, which includes considering: •

How well the project met its objectives;

•

How well the project applied value for money principles of effectiveness, economy, efficiency in relation to delivery of its outcome;

•

What has happened because of DFID funding that wouldn’t have otherwise happened; and

•

How well the project aligns with DFID’s goals of supporting the delivery of the SDGs.

The following methods were adopted to ensure this objective was achieved. How well the project met its objectives: The logframe clearly outlined the expected impact, outcomes and outputs from the project. The annual project reports were verified to identify results that were achieved using the indicators as the means of verification of expected outputs. The review also verified that the activities set out to be implemented were achieved. The milestones and the targets outlined in the logframe were also verified to determine how many were achieved. Annex VII shows the outcome of this verification. Evaluation of Value for Money: Guidelines provided in Annexe 1 of the ToR which clearly outlined UK Aid’s guidelines for determining value for money was used to verify how the project performed in this aspect. Documents, processes and systems that were available were reviewed to determine how well the project applied value for money principles of effectiveness, economy, efficiency in relation to delivery of its outcome. The finance manager, the CHISHPIN project manager, the UPprogramme manager and Country Director (CD) were interviewed to get clearer understanding of processes, and systems in place. The review determined; •

Types of cost incurred and the relevance of activities or services for which they were incurred.

•

Were expected results achieved from the activities carried out as a result of the cost incurred?

•

Did target communities get the expected benefits as a result of the cost incurred in the project?

•

Were there changes to proposed cost and what were the drivers for such changes?

•

What systems and/or processes are in place to determine and ensure that the right price was obtained and utilized – e.g. in procurement processes?

•

Verification of cost effectiveness in relation to inputs, outputs and outcomes.

It was time consuming to verify all of these on 100% basis, so samples were verified based on some activities carried out. What happened because of DFID’s Funding? To find the answers to this question the following evaluation was done: •

Verification of baseline data against current achievement of outputs and outcomes through review of reports and other forms of documentations

•

From responses to questionnaire administered in households in the sampled communities 13

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report •

Testimonies from focus group discussions (FGD), interviews in schools, LGA WASH departments, and Local Government Council comparing these with discussions with UP programme staff.

•

Environmental walk and structured observations

Alignment with DFID’s goal of supporting the SDG’s: The goal is poverty reduction based on community’s definition of poverty. UK Aid Direct focuses on reaching the most vulnerable and marginalised populations, girls and women, to ‘leave no one behind’. •

Key question addressed was: How has this project improved wellbeing of the beneficiaries? Any positive behaviour changes in targeted groups as a result of the project interventions?

•

FGD was used to elicit benefits poor community members have seen since program started

•

Questionnaires was used to elicit possible changes from baseline conditions

To be able to identify the unique characteristics for United Purpose performance in the delivery of LGA wide ODF as compared to other CLTS practitioners in Nigeria. If there are any substantial differences, (possible explanations should be provided). UP has already demonstrated that it is a pace setter in the arena of CLTS implementation in the Nigeria. To identify the unique characteristics of UP’s performance; •

Current approaches adopted by other practitioners of CLTS were contrasted with UP’s current practices as implemented in the CHISHPIN project

•

Interviews were carried out with some identified key informants from active CLTS implementing organizations like UNICEF, WaterAid, NTGS, Project Advisory Committee (PAC) to understand their current practices.

2.1.1 Sample Size CHISHPIN project CLTS communities are 1,180. For this evaluation a sample size of 118 communities was agreed based on the population of 1,180 communities. This represented 10% of the total communities of 1180. The breakdown of the proposed sample communities by ward and LGA is shown in table 1. Table 1:Proposed Sample Communities distributed by ward and LGAs LGA

Biase

Ikom

Yala

Total

Wards

No. of CLTS Communities

335

393

452

1180

10% per LGA

118

Communities for verifying Population

108

Proposed Households to be Sampled

1,006

1,432

1,431

3,869

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

2.2 Strengths and weaknesses of selected design and research methods Going by the design of the CHISHPIN project it was determined that it will not benefit from a counter factual evaluation approach because it has not been designed as an experimental or quasi-experimental project from the onset. Applying this approach will therefore be very difficult to justify. A contributory-based approach was therefore adopted. Looking closely at the design of the project from the proposal document, the project was designed to answer the standard indicator related to MDG 4B: "Number and percentage of under-fives who have: a) experienced one or more episodes of diarrhoea in the last two weeks and; b) received ORT." CHISHPIN project set out to improve hygiene behaviour of people in 1,180 communities with the intent to impact an estimated 91,300 under-fives vulnerable to deadly diarrhoeal diseases transmitted by inadequate sanitation and hygiene, approximately 351,000 women and girls whose lack of sanitation access means disproportionate time and resource expenditures, physical insecurity, and feelings of indignity in addition to other vulnerable groups. It is also expected that an estimated 149,000 girls will further benefit from private latrines at schools which will enhance their educational enrolment, retention, and performance. The logic and assumption here are that the CHISHPIN project will produce these results amongst others. There are however other interventions that can lead to reductions in diarrheal disease. The World Health Organization identified the following interventions as possible to reduce diarrheal disease. These have been adopted in this evaluation to constitute the ‘causal package’ • • • • • • •

access to safe drinking-water; use of improved sanitation; hand washing with soap; exclusive breastfeeding for the first six months of life; good personal and food hygiene; health education about how infections spread; and rotavirus vaccination.

This evaluation took into consideration these contributory interventions and determined how many of these were ongoing in the project environment. This evaluation therefore adopted the contribution-based approach because "they are helpful for overcoming the attribution issue of proving cause and effect." The decision to use this approach was guided by Stern et al. (2012) who said, "The notion of a ‘contributory’ cause, recognizes that effects are produced by several causes at the same time, none of which might be necessary nor sufficient for impact." In this case the evaluation wanted to determine if the CHISHPIN project “was a necessary ground preparing cause, a necessary triggering cause or something that did not make any difference and a similar effect would have occurred without the intervention.” (Stern et al, 2012, p, 40 41). The CHISHPIN project is conceptualized as a one-cause intervention associated with one outcome or impact. In a simplified form, the logic is that facilitating behaviour change in communities that leads them to construct latrines and handwashing facilities and stop defecating in open, will reduce under-five diarrhoea by 10%. The design of the evaluation took this conceptualization into consideration. This is because different designs and methods are best able to deal with different kinds of causal relations.

2.3 Summary of problems and issues encountered Community mobilisation is very key to the success of any interaction with community members. It was challenging to effectively mobilise communities for household interview, FGDs and KII. 15

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report There were instances where community people refused to come out for meetings because they felt they were not given adequate notice. In other instances, most adults in some communities were said to be away at farm and would not be back for weeks. Because of these limitations some of the respondents targeted for households, FGD and KII could not be reached. The proportion of targeted households reached was about 1054 in Yala, 888 in Ikom and 599 in Biase LGA. Due to the tight schedule for an end of project evaluation like this one and the depth of inquiry required by the ToR, enumerators had to work on weekends, and this meant schools and health centres could not be reached in locations where surveys held on these days. In Ikom and Biase, key LGA officers like the Head of Local Government Administration (HOLGA) and their Director General Services and Administration (DGSA) were not available to be interviewed at it was not possible to go back to the LGAs, but CLTS information for that category was collected from the Directors in Charge of the Environment Department and the WASH Coordinators. Population verification exercise had to be combined with household interviews to save cost of duplication of enumerators in the exercise. The challenge this brought, was that enumerators could not stay too long in households to do physical count of members or wait indefinitely for households that are not available at the first instance. They relied on the information provided to them by household heads. Only households that were available at the time of visit were interviewed. In situations where it was necessary to cycle back to confirm information that might have been missed the previous day, this was done. This could however not be done in cases where the households were not available. The aspect that this affected most is the data for verification of population. An alternative data collection approach was therefore adopted to ensure a more robust population information was collected. Effort was made to evaluate the health data on diarrhoea in community health centres. Where there was record, they were not properly grouped, such that there was no data for under-fives except in two out of the 18 health facilities that were reached. Data from the Epidemiology Unit was collected, and this was also used. Finally, not all CLTS communities had health centres and schools located within them. So, to avoid double counting, communities already counted by one group, enumerators were informed to only collect information from schools available in the communities they visit as part of the sampled communities.

3.0 Findings 3.1 Overall Results This section outlines the general findings from the interview of households, FGD and KII with community people, and KII with institutions at national, State and LGA levels. It does not present an in-depth analysis or explanation; this is done from section 3.3 where further analysis and explanations of findings are done in response to evaluation questions. The results of the field work showed that, instead of the 3,869 households proposed to be sampled, only 2541 households responded. This represents 66% of the proposed sample households. Figure 1 shows the distribution of households per LGA. In Yala 1054, representing 74% of the planned were reached, Ikom 888 or 62% of planned were reached and in Biase 599 or 60% was reached.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

Number of Households in the LGA

Households per LGA 1200 1000 800 600 400 200 0 Biase

Ikom

Yala

Project LGAs

Figure 1: Number of Households reached per project LGA 76.5% of the respondents in the households visited are Farmers, Civil Servants are 10.2% percent, while those in business are 4.4% as the figure 2 shows.

Number in each Category

Types of Occupations in Communities 1945 2000 1800 1600 1400 1200 1000 800 600 400 200 0

259

111

Occupation Category

Figure 2: Occupational distribution among the respondents .

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Two types of latrines were predominant in the sampled HHs as shown in figure 3.

Types of Toilets 4%

26%

Pit Water system

70%

Unspecified

Figure 3:Types of Latrines in Sampled Households Householders were asked about the challenges they have with handwashing. 33.9% of those that responded said they do not have enough water, 3.5% said they could not afford soap while 2.6% said they find it difficult to get ash. Since water was not provided through the CHISHPIN project, the evaluation wanted to understand if water would be a challenge to households or not. Households were asked about their source of water, 67.5% said they got their water from streams, while about 14.6% got theirs from borehole and 10% from well. This result is in line with the findings from the Baseline which found surface water to be the predominant source of water supply for the communities in the three LGAs. While not checked in the evaluation, the baseline found that the communities use this surface water for drinking and cooking.

Community Water Source

10%

Stream Borehole

15%

Well

67%

Unspecified

Figure 4 Community Source of Water

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report 61% latrines observed have drop hole cover, while only 0.3% did not have as the figure 6 shows. The number of latrines with cover is an indication that the community households imbibed the hygiene and sanitation messages they received as a result of the intervention.

Presence of Toilet Hole Cover

39%

Yes

61%

No Response

Figure 5 Percentage of Latrines with Drop Hole Cover 12.5% of the latrine observed had faeces on the floor, while 22.2% did not. There was a high no response rate for this question.

PRESENCE OF FAECES ON LATRINE FLOOR faeces on the floor of latrine

% 1640

558

315

65.26

22.20

12.53 Yes

No Response

Figure 6 Latrines observed with Faeces on the Floor

Types of Super Structures 201

8.0

BRICKS

624

24.8

ZINC

698

27.8

697

288 11.5

RAFFIA

MATS

Type of Super Structure

Figure 7 Types of Latrine Superstructure Materials 19

27.7

CLOTH %

0.2

NO RESPONSE

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Zinc (24.8%), Raffia (27.8%) and Cloth (27.7%) are the main materials used as latrine super structures in households visited. The predominant use of low cost materials as superstructure for latrines is consistent with focus of the CLTS approach which is more on effectively containing the faeces and not on the beauty of the facility.

Number of Responses

Number of Handwashing Facility by Toilet 1500 1000 500 0 YES

NO RESPONSE

Respones Handwashing facility

Figure 8 Number of Handwashing Materials Observed 47% of the compounds observed had handwashing facilities by their latrines while 6.4% did not as figure 9 shows. This is an indication that communities imbibed the hygiene and sanitation messages from the project.

Number of Responses

Presence of Soap and Ash 1400 1200 1000 800 600 400 200 0 Yes

No Response

Responses Soap/Ash present

Figure 9 Presence of Soap or Ash by Handwashing Station 35.2% of household observed had soap or ash at the handwashing station while 10% did not as figure 10 shows. 49% of households observed seem to have evidence of use of handwashing facilities recently while 12% did not. Evidence used was observing any sign of fresh presence of water around the station. This is not a strong indication that the stations are not being used though. The high rate of no response may also be as a result of enumerators not wanting to make a committal statement when not sure. 20

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

Sign of Recent use of Hanwashing Facilities 39%

49%

Yes No

12%

No Response

Figure 10 Use of Handwashing Facilities

Faecal Matter Outside Compound 4% 24%

yes no no response

72%

Figure 11 Presence of Faecal Matter around the Compound 24% of household observed did not have faeces outside their households while 4% had. Again, there was a high percent, 72% of no response. The absence of OD around the houses confirmed the change in behaviour in the community people.

Schools with Latrines 2% Yes No

98%

Figure 12 Schools with Latrines and those without

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report During the FGD with school children they were asked if they have latrines in their schools, from their responses it shows that 98% of the schools have toilets while 2% said no.

Environmental Health Clubs in Schools

32%

Yes

68%

Figure 13 Schools with and without EHCs Not only was there a high number of latrines but the School children (particularly girls) said the latrines met their needs as women, when asked. 68% of the schools reached did not have Environmental Health Clubs (EHC) while 32% have. The responses were received from school children both boys and girls. Disparity between schools with latrines and those with EHCs seems to be much. It is not clear if the children have a good knowledge of how EHCs function or not. This might have affected the results. One of the strategies used by the CHISHPIN project to promote and inculcate in the school children effective hygiene and sanitation habits is the use of the Global Handwashing Day (GHWD) celebrations.

Number of Responses

School Participation in GHWD Celebrations 100 80 60 40 20 0 Yes

Responses Number

Figure 14 Schools that Participated the Global Handwashing Day events organised by UP When asked if they participated the GHWD events organized by UP, 82%of schools interviewed said they did, while 18% said they did not. Asked â&#x20AC;&#x153;how did this annual event (The GHD) help you

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report as a student and as a school?” The students respondent as in table 2; It will appear from these responses that the children learnt a lot from the GHWD events. Table 2 Responses from School Children on Benefit of the GHWD celebrations How did this annual event (The GHWD) helped you as a student and as a school?

It teaches me to wash my hands always; creates the consciousness It helps me to understand the basic needs of living a hygienic lifestyle We now know that we must wash our hands whenever we come out of latrine We know that handwashing prevents a lot of children diseases It helps to stop the spread of communicable disease It helped me to always remember to wash my hands Since they came each pupil has now imbibe the culture of handwashing. We now have a basin of water with soap in the latrines We now know that it is compulsory to wash our hands after using the toilet I am now aware that most diseases can be contracted through the hands It has changed the attitude of my friends and I, we now wash our hands each time we finish playing, after using the toilet and many other times

Annex VIII provides the summarised results of the different FGDs and KIIs that were done with the communities. The evaluation sought information on the effect of CLTS implementation on children under-five years in health facilities across the 3 focal LGAs. 18 health facilities were reached. In those reached, no targeted information (on under-fives) was available except in Lutheran health centre – Wanikade, Yala LGA (where the data was not segregated by gender), and Health centre, Nde – Ikom LGA.

UNDER FIVE DIARRHEAL CASES IN WANIKADE YALA LGA Response

50 33 17 3 2015

2016

2017

2018

Years

Figure 15 Under-five Diarrheal cases reported in Wanikade Health Centre in Yala LGA The charts (Figures 15 and 16) show the rate of decline of diarrhoea in children less than 5 years in Lutheran Clinic, Wanikade, Yala LGA and Nde – Ikom LGA using 2015 as the base year. Although the data used came from a single facility in each LGA (Ikom and Yala), they show a steady decline in the attendance of under five children to health facility from 2015 with a total of 13 to 0 in 2018. 23

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

Under-Five Diarrheal Cases in Nde Health Centre Ikom LGA Number of Cases

52 28 20 13

2015

2016

2017

2018

Years

Figure 16 Under-five Diarrheal cases reported in Nde Health Centre in Ikom LGA

Number of reported Cases

Under-five Diarrheal cases reported in Baise LGA 100 80 60 40 20 0 JAN

FEB

MAR

APR

MAY

JUNE

JULY

AUG

SEPT

OCT

NOV

DEC

Months 2016

2017

2018

Figure 17 Biase LGA Under-five Diarrheal cases reported to State Epidemiology Unit

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

Number of Reported Cases

Under-Five Diarrheal Cases in Reported in Ikom LGA 140 120 100 80 60 40 20 0 Jan

Feb

Mar

Apr

May

June

July

Aug

Sept

Oct

Nov

Dec

Months 2016

2017

2018

Figure 18 Ikom LGA Under-five Diarrheal cases reported to State Epidemiology Unit

Number of Cases Reported

Under-Five Diarrheal Reported Cases in Yala LGA 20 15 10 5 0 Jan

Feb

Mar

Apr

May

June

July

Aug

Sept

Oct

Nov

Dec

Months 2016

2017

2018

Figure 19 Yala LGA Under-five Diarrheal cases reported to the State Epidemiology Unit Monthly health data, between 2016 and 2018, reflecting diarrheal in under-five children, obtained from the State Epidemiology Unit in Calabar, shows a close relationship with the trend obtained with the data from two health centres. This data ought to have been more reliable because it is based on monthly returns by the 35 health centres in the State While these show general indication of a downward trend they are not enough evidence upon which to base contributory causal inference, it is however an indication that such a pattern of downward trend may exist in the communities where the project was implemented. . Figure 20 shows a summarized version of 2016, 2017 and 2018 reported cases for the three LGAs. It will appear from this graph that, on average, there was downward trend in in three LGAs.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

Number Cases Reported

Yearly Summary of under-five Diarrheal Cases reported in the three LGAs 500 400 300 200 100 0 2016

2017

2018

Years Biase

Ikom

Yala

Figure 20 Overall LGA Under-five Diarrheal cases reported to State Epidemiology Unit Three objectives were set for the CHISHPIN end of project evaluation as itemized in section 1.2 above. This evaluation can confidently confirm that the objectives were achieved as explained below; A close review of CHISHPIN project annual reports for 2016 – 2019, as well as the project Logframe verifies its record of achievement. The evaluation was carried out in the 118 communities in the three LGAs. The number of households the evaluation planned to reach was 3,869 but 2541 was reached. The outcome from the household, KII, FGD interviews carried out showed that CHISHPIN project succeeded in facilitating true behaviour change in the communities in the intervention LGAs. All 1180 CLTS communities were triggered, with 100% of the communities in two of the three LGAs now ODF, practicing new hygiene and sanitation behaviours of handwashing with soap and ash at critical times, covering their latrines to prevent flies, effectively managing children faeces as the findings already reported above show. A review of achievement against logframe milestones and targets was done. This revealed that CHISHPIN project team achieved its milestones and set targets within the three years of its implementation. One of the expectations of the evaluation under objective 1 is to “Verify target population through selective sampling in the three LGAs.” This was done in 108 communities in the three LGAs. The result of the evaluation is presented on table 3 for Biase, table 4 for Ikom and table 5 for Yala LGAs. The tables compare population figures collected by UP during the implementation of the CHISHPIN project through a peer review process and documented in files side by side with community figures. The evaluators made use of community data collected over time through the development of community commitment list. The commitment list is a register of community members in each household that enables the WASHCOM to determine the number of latrines a community should construct to meet its population needs. This register is developed by the WASHCOM through interaction with each household in the community and kept updated. For example, we found a register with a household name struck out because they had left the community and, in another instance, because the householder had passed on.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report The result of the exercise shows that in some instances the figures obtained by UP is in exact agreement with the community figures, in most others they are very close with marginal differences. There are some large variations in the differences with the highest being 419 in Biase, and the lowest being negative 185 also in Biase. These extremes are however few and could be counted as outliers.

Diff in Total comm Pop

Population Total (Evaluation)

Community Population (UP)

CLTS Community

Council Ward

LGA

Table 3: Showing the Population Verification of Biase LGA

Biase

Adim

Obioko 2

255

Biase

Adim

Ijiman 2

228

Biase

Adim

Ijiman 1

-13

Biase

Agwagune/Okurike

Akugom

150

178

-28

Biase

Agwagune/Okurike

Usaja

209

202

Biase

Agwagune/Okurike

Eke-Obo 1

177

Biase

Abayong

UWOM EBA EKPE UNOR

Biase

Abayong

OKWEKWE

Biase

Abayong

ABAMBA

175

Biase

Ikun/Etono

Ebangha 4

120

303

-183

Biase

Ikun/Etono

Ebangha 3

185

-93

Biase

Ikun/Etono

Bisu

Biase

Ikun/Etono

Agana

302

120

182

Biase

Ikun/Etono

Omobe 1

182

127

Biase

Ikun/Etono

Obum

498

419

Biase

Biakpan

Ano Obia

Biase

Biakpan

Ano Obu 2

130

230

-100

Biase

Biakpan

Ano Okot

179

169

Biase

Umon North

Ikot Okpora Village A

218

136

Biase

Umon North

Ikot Okpora Town A

218

274

-56

Biase

Umon North

Ikot Okpora Town B

115

309

-194

Biase

Ehom

New Heaven

272

322

-50

Biase

Ehom

Odoro

485

130

355

Biase

Ehom

Betem J

301

170

131

Biase

Akpet/ Abini

Ikpariyong

290

255

Biase

Akpet/Abini

Abeyere

165

258

-93

Biase

Akpet/Abini

Anekpe

251

Biase

Akpet/ Abini

Okochere 2

145

Biase

Akpet/Abini

Tower 2

130

135

-5

Biase

Akpet/Abini

Usari

125

Biase

Akpet/Abini

Ossine 4

105

195

-90

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Biase

Umon South

Agbagana

119

105

Biase

Umon South

Inyene Odukusung

6087

5460

627

Total

Nkutal

512

697

-185

Ikom

Abanyom

Licy Alibankang

440

394

Ikom

Abanyom

Ekumaya

146

157

-11

Ikom

Akparabong

Nturokim 4

630

600

Ikom

Akparabong

Okwa

332

361

-29

Ikom

Ikom Urban 1

Bokomo

503

462

Ikom

Ikom Urban 1

Kalime 3

955

800

155

Ikom

Ikom Urban 1

Cameroun Street

628

616

Ikom

Nnam

Emanghesha

189

153

Ikom

Nnam

Nlul-Emang

149

142

Ikom

Nnam

Etak-Egboja

222

220

Ikom

Nde

Anakan Egba (Sch. Road)

518

Ikom

Nde

Lagos

337

266

Ikom

Nde

Army Barrack

475

250

225

Ikom

Ikom Urban 2

Osung

986

980

Ikom

Ikom Urban 2

Nkamafor

640

540

100

Ikom

Ikom Urban 2

Akrip

513

Ikom

Ofutop 1

Okangha Nkpansi 1

145

-63

Ikom

Ofutop 1

Omindom 1

172

210

-38

Ikom

Ofutop 1

O.N.EleMba 1C

221

289

-68

Ikom

Ofutop 1

Ele Mba 2

143

233

-90

Ikom

Ofutop 2

Okammolan

199

Ikom

Ofutop 2

Osusumkpor

509

324

185

Ikom

Olulumo

Okokoma-Olulumo

122

124

-2

Ikom

Olulumo

Down Effi

289

Ikom

Olulumo

Erim Obot MP

568

600

-32

Ikom

Nta-nselle

Lebenjork

170

216

-46

Ikom

Nta-nselle

Ikpatala

256

125

131

Ikom

Nta-nselle

London

229

230

-1

Ikom

Yala-Nkum

Okpochi I

192

170

Population Difference

CLTS Community

Abanyom

Total Population Reached

Council Ward

Ikom

Community population

LGA

Table 4: Showing the Population Verification of Ikom LGA

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Ikom

Yala-Nkum

Atimaka I

206

337

131

Ikom

Yala-Nkum

Opu-Owuwe

293

206

11,826

11,254

572

Total

Population Differential

Total HH Population (Evaluation

Community Population (UP)

CLTS Community

Council Ward

LGA

Table 5: Showing the Population Verification for Yala LGA

Yala

Ugaga

Jamaica

304

245

Yala

Ugaga

Opopu-Ipuole

156

153

Yala

Ugaga

Ikpachor

317

343

-26

Yala

O'OH

Yakpla III

383

248

135

Yala Yala

O'OH O'OH

Idiku II Court Community

247 170

302 301

-55 -131

Yala

Ijiraga

Ngomo 1

231

208

-77

Yala

Ijiraga

Ngomo 2

129

128

Yala

Ijiraga

Bituol-Ijiraga

193

184

Yala

Okpoma

Opihuorh-Okpudu

238

206

Yala

Okpoma

Aduoje

299

223

Yala

Okpoma

Utu - Agu

250

235

Yala

Yahe

Enugu/Owerri

244

137

107

Yala

Yahe

Apata /Woleche

224

200

Yala

Yahe

Abakalike Road A

249

328

-79

Yala

Okuku

Imaje Road New layout

389

403

-14

Yala

Okuku

Ocheyenor Left 1

521

306

215

Yala

Okuku

St. Joseph lane 2

959

861

Yala

ECHUMOFANA

ETIEKPO 1

279

199

Yala

ECHUMOFANA

ETIEKPO 2

291

192

Yala

ECHUMOFANA

IBI 1

237

247

-10

Yala

YACHE

ETULOKO

305

199

106

Yala

YACHE

OBIAJE 1

340

266

Yala

YACHE

OBIAJE 2

213

195

Yala

Gabu

Ijaku/Ogboku

328

205

123

Yala

Gabu

281

185

Yala

Gabu

Onye unity Adikpe/Onyeada Utu Oke

294

290

Yala

WANOKOM

AHAKABENE

156

126

Yala

WANOKOM

ITIGIDI

164

240

-76

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Yala

WANOKOM

UGAGA

151

Yala

Mfuma/Ntrigom

Agba

237

217

Yala

Mfuma/Ntrigom

Mission

256

258

-2

Yala

Mfuma/Ntrigom

Elu

159

106

Yala

IGEDE

Ukwekwe/Otam 1

100

-22

Yala

IGEDE

Ukwekwe/Otam 2

194

193

Yala Yala

IGEDE Wanikade

Ofema/Eshum Okpodokem/Ladi/Ehi

99 242

87 193

12 49

Yala

Wanikade

Ugburu/Calabar

176

291

-119

Yala

Wanihem

Owele Bansara

547

Yala

Wanihem

Adum Lema Ukelle 1/Adum Lema Ukelle 2

546

587

Yala

Wanihem

Ezokobe Tokorsala/Laddi/Ugwaba/Igbe Total

239

302

-63

11,315

10,786

529

Table 6 Summary of Population Verification for the three LGAs United Purpose Figures Yala Biase Ikom Total

Evaluation Figures 11315 6087 11826 29228

Grand Variance (people)

10786 5460 11254 27500

529 627 572 1728

Table 6 is a summary of the population verification results at each LGA. It shows that overall the variance between the population figures obtained by UP, through the peer review process, and the result obtained by the evaluators from the community commitment register, is 1,728 people. This variance is about 5.9% of the UP population figures. This is not a very significant difference and confirms that the population figures used by UP for implementing the CHISHPIN project is reliable. The Evaluation also set out to assess the extent to which the project was good value for money. Section 3.4 provides a detailed explanation on how CHISHPIN project achieved this objective. The third objective the Evaluation set out to achieve is to â&#x20AC;?Identify unique characteristics for United Purpose performance in the delivery of LGA wide ODF as compared to other CLTS practitioners in Nigeria.â&#x20AC;? UP is more effective in its approach to reach ODF status for LGAs, as compared with other organizations. For example, UP has achieved 4 LGAs declared ODF in CRS with 1 awaiting

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report formal declaration by NTGS, against 5 ODF LGAs for UNICEF (FGN-UNICEF programme) which is working in some 120 LGAs. Moreover, FGN-UNICEF as well as WaterAid started much earlier than UP. They started implementing CLTS since 2006. The uniqueness of CHISHPIN project approach was attested to by UNICEF and the NTGS during this evaluation. According to the UNICEF WASH (Sanitation) specialist at the National level, CHISHPIN project approach “is strategic”, He is “impressed with the community driven process. Community dialogue is excellent” .”Their documentation is very good.” “CHISHPIN approach ensures that community is at the forefront and this is excellent.” “There is also strong and good community dialogue. Grassroot engagement is excellent.” Asked if he would recommend CHISHPIN project approach for adoption at national level, he had this to say, “The community engagement approach of CHISHPIN can be adopted at National level. UP CLTS is grassroot and cheaper”. An NTGS member interviewed also presented his views of the uniqueness of CHISHPIN project in the sector compared to other similar organizations. According to this NTGS member, “UP strategies are completely different from that of other organizations in the sector.” . The approach UP adopted is what they (NTGS) had advised UNICEF more than 3 years ago but they did not accept it. “UP uses CSO plus LGA at grassroot level to create better ownership and total commitment at community level. UP strategy engenders more commitment, has a sense of responsibility and ownership, facilitates better community buy-in and has peer influence.” These are clear testimonies from national level sector players confirming the uniqueness of UP’s CLTS approach.

3.2 Assessment of accuracy of reported results. This evaluation has a clear logic of enquiry. The questions posed are very clear and designed in response to the objectives of the evaluation and the evaluation questions. From the field data on FGDs, KII, HH and Observations, it is clear that implementation of the CHISHPIN project changed the Sanitation and Hygiene behaviours of the target communities. This is reflected in their testimonies and practices observed across especially in Yala and Ikom LGAs. For example, all men interviewed during the FGDs in Yala and Ikom LGAs, when asked if they have latrines, they all said they have. Not one person said no. This however was not the case in Biase, particularly in communities that were not yet ODF. These results support the reported 100% ODF in Yala and Ikom. Community members (women, men, teachers and pupils) were all unanimous in their comments about the benefits of the CHISHPIN project to them. They expressed happiness at their transformed lifestyles for example, they no longer use cross bars for defecating; they no longer go to the bush to defecate in the open; they no longer perceive smells of shit all around as was the case before the CHISHPIN project started; they no longer have to contend with flies that perch on their shit and then come and perch on their food as a result they no longer experience cholera episodes; they spend less on hospital bills; and the children (especially girls) do not have to skip school. Furthermore, the community can now plant vegetables and they can also pick fruits that drop from trees, wash and eat without worries of the fruits being contaminated with faeces. Community members also demonstrated a good understanding of sanitation and hygiene behaviours like keeping their latrines always clean and covered; effective handwashing with soap or ash at critical times; good management of children excreta and keeping their environment clean. From the desk review of project reports, the evaluation confirmed that reports of UP (Activity Monthly, Annually) measures progress through the Monitoring and Evaluation compilation. Interview of CSOs, WASH departments, and beneficiary communities confirmed UP reporting as accurate. Up reported that they developed 2 Compendiums and 2 newsletters and shared with partners and water sector players. Evidence of these documents were verified and discussion 31

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report with UNICEF WASH Sanitation specialist confirmed that UP programme was good with documentation of learning. Testimonies from beneficiary communities interviewed during this evaluation supports claims made by UP in its reporting. The evaluation confirmed that the Communities have stopped defecating in the open and the use of cross bars for defecating was stopped by the communities when they realised that this practices were affecting them negatively (socially and health wise). UP reported building the capacity of WASH staff and CSOs. Discussion with CSOs and WASH department staff confirmed that their capacity was built. The unprecedented results achieved by the WASH department and CSOs staff that facilitated CHISHPIN project, is a clear indication of their grassroot level CLTS facilitation skills, which has been commended by sector actors interviewed during the cause of this evaluation.

3.3 Relevance The CHISHPIN project was designed to answer the standard indicator related to MDG 4B: "Number and percentage of under-fives who have: a) experienced one or more episodes of diarrhoea in the last two weeks and; b) received ORT." The project is also aligned to the Sustainable Development Goals (SDG) 3 and 6. The project supports the achievement of SDG 3 whose goal is to: “Ensure healthy lives and promote well-being for all at all ages. Specifically, by 2030, end preventable deaths of newborns and children under 5 years of age, with all countries aiming to reduce neonatal mortality to at least as low as 12 per 1,000 live births and under5 mortality to at least as low as 25 per 1,000 live births.” While it was not possible to obtain sufficient data to confidently establish a causal inference that support the impact of ODF and effective handwashing on the health of the communities, and the project did not collect monitoring data that could be used, results from two out of the eighteen health centres reached (one in Yala and the other in Ikom LGA) that were specific to children under five, shows a trend of decline in diarrhoea cases reported between 2015 and 2018. Health data was also collected from the State Epidemiology Unit in Calabar. It showed that there was an overall reduction in the number of under-five reported diarrheal cases between 2016 and 2018. While these are not sufficient to completely attribute the reduction in reported diarrheal cases to the CHISHPIN intervention, it is a strong indication of the intervention being a necessary triggering cause. FGD responses from the men, women and school children groups indicated that the communities experienced improvements in their health, when asked what they saw as the benefits of the CHISHPIN project? Table 4, 5, 6, 7, and 8 provide a cross section of their responses. The testimonies were overwhelming pointing to the fact that community members experienced health benefits ranging from reduced hospital bills because of reduction in diarrheal disease among their children to absence of cholera outbreak since the CHISHPIN project commenced. The project has contributed about 14 % to the Cross River State strategy to end Open Defecation in 2023 by successfully eliminating open defecation in two out of its three focal LGAs and more than 50% in the third. In the total of 5 LGAs where ODF have been celebrated in Cross River State, 2 are direct CHISHPIN project LGAs. This has further strengthened the state’s commitment to ensure a scaling up of CLTS activities in the remaining 13 LGAs. There is a possibility to adopt CHISHPIN implementation strategies as key success strategies in the sector by other organizations in the ongoing development of the national plan to deliver an ODF Nigeria by 2030. The CHISHPIN project is described as excellent grass root approach by key stakeholders. This is the closest result based influence the project might have on Sanitation policy.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report The CHISHPIN project has no doubt assisted the State Agency for Rural Water Supply and Sanitation (RUWATSSA) in meeting its mandate and also helped UP to carve a niche in the Sanitation Sector in Cross River State. Table 7: Extracts of FGD health response from the women group As women, what do you see as direct benefits of the project to you?

Children sickness reduced. Rate of illness in adult reduced. Reduced visit to hospital Also helped us not to contact diseases Our children and adults can now pick fruits freely It has helped eliminate some diseases that we have hitherto considered are caused by witchcraft Flies carrying diseases like cholera have been reduced, our food is now safe from contamination Illnesses especially amongst children has reduced tremendously Most of the infectious diseases in women have reduced because we are no longer using the public toilet There used to be outbreaks of Cholera, dysentery, diarrhoea, malaria and children use to die in their numbers but all that has stopped now It has helped women save their money which they would have used in buying drugs for themselves and their children Many diseases are no longer rampant We no longer have the infections we use to have when we didnâ&#x20AC;&#x2122;t know how to properly clean the toilet

Table 8:Extracts of FGD health response from men group What do you see as the benefits of having latrines in your household and in public places?

Also, the rate of illness amongst children has reduced The benefits are the health of my family and strangers that visit me It has reduced the plague of cholera due to the reduction of flies; I have not treated diarrhoea since the project Many diseases have been drastically reduced or eliminated Cholera has not been reported for a long time, the community is now cleaner

Table 9:Extract of FGD health response from School Children What do you see as the benefits of the CHISHPIN project to your school and to yourself?

CHISHPIN has taught us hygiene proper It brings about convenience and healthy living It keeps me from contacting disease and from danger I have benefitted from schooling in a disease-free school It reduces the spread of diseases It reduces many diseases associated with faeces and trouble flies

Table 10:Extract of KII health response from Traditional leaders What do you see as the specific benefits of the project intervention to your people?

The project has eliminated cholera, diarrhoea, flies and air pollution. No more bad odour There are less sicknesses The environment is clean and free from illness caused by dirt It has made me proud having a toilet

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Table 11:Extracts of KII health response from School Teachers

6. What benefits have the CHISHPIN project brought to the school since it started?

Awareness in hygiene has reduced disease and infection It has made the children to be very hygiene conscious School attendance has not dropped recently Sickness based absenteeism has reduced Pupils now understand the benefit of washing their hands at Teachers claim that students no more complain of sickness, and are not absent; -

CHISHPIN project also supports the achievement of SDG 6 with the goal to: â&#x20AC;&#x153;Ensure availability and sustainable management of water and sanitation for all. Specifically, by 2030 achieve access to adequate and equitable sanitation and hygiene for all and end open defecation, paying special attention to the needs of women and girls and those in vulnerable situations.â&#x20AC;? The responses to Household (HH) questionnaires shows that 82% of respondents in the three LGAs have access to their own latrines. 2 LGAs out of the total of 18 LGAs i.e. two (2) LGAs (Ikom and Yala) are 100% ODF, while more than 54% of Baise communities are ODF. This represents 13.9% contribution to the ODF target of Cross River State (CRS) from the perspective of the Local Government Areas. This is a significant contribution to the realization of an ODF State by one project. By this achievement CHISHPIN project has made significant input to the SDG target of CRS as outlined by SDG 6. In terms of population of the poor and marginalized people reached, the CHISHPIN project provided social, economic and health benefits to 358,886 people in the three LGAs with access to latrines and handwashing facilities. Based on the outcome of the evaluation, CHISHPIN project mainstreamed gender equality into the project. Table 12:Women involvement in the implementation of CHISHPIN project How have women been involved in this project?

Involved by contributing to the building of latrines Carrying out inspection of latrines Organizing sensitization twice monthly Women involved in building toilets As women we come together to talk to each other about the benefit of having and using toilets, some women joined WASHCOM We are involved as agents of transformation in the community Being the wife of the Chief of the Community I am involved in leadership and further training of our children and women Sometimes we contributed and give money to our husbands for the buying of materials, like zinc, nail and cement The maintenance of toilets is done by women Women put pressure on their husbands to construct latrines as some men testified

Another aspect of the CHISHPIN project that demonstrates the mainstreaming of gender equality in the project is in the membership of Water and Sanitation Committees (WASHCOM), which is the community level team responsible for the management and leadership. A summary of the distribution of women and men in the constitution of WASHCOM per LGA is shown in figure 22. It clearly demonstrates that CHISHPIN project carried the women along in the management of the intervention processes.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Female WASHCOM executive members were trained as CLTS facilitators and were actively involved in the triggering process across the LGAs. Some men specifically identified the womenâ&#x20AC;&#x2122;s role as being the ones to keep the latrines clean. Both men and women openly confirmed they were happier to own and use household latrines for privacy and prestige. That they believe has saved them from a lot of embarrassing issues like looking at their nakedness and being able to defecate conveniently. Women also mobilized their men to construct household latrines in many communities in Ikom and Yala. This was confirmed, with gratitude to their wives, by men interviewed in their group. It is also reported that in schools, separate toilets were built for male and female. The schoolgirls interviewed expressed their satisfaction with their latrines saying overwhelmingly that the latrines met their needs as women.

Gender Distribution in WASHCOM Executive Biase

Ikom Yala 0

Men

100

120

140

Women

Figure 21 Gender Distribution in WASHCOM Executives in the Project LGAs

3.4 Effectiveness A review of CHISHPIN project annual reports from 2016 to 2019 provided a basis for evaluation of the record of achievement of results. CHISHPIN logframe had milestones and targets against the outcomes and outputs with specified time of delivery. Annex VII provides a table that compares each milestone and target to the achievement. The table shows that in year one and the earlier part of year two about 20 milestones and targets were not achieved and about 5 were moved. In the later part of year two and year three however the team was able to bounce back, despite the suspension of activities for five months, to achieve and in some instances to exceed the milestones, eventually achieving LGA-wide ODF status in two out of the three LGAs. The evaluation found that due to non-payment of Government counterpart funding agreed at the onset of the project, UK Aid Direct, funders of the project, suspended the project for five months. The CRS Government 25% contribution was expected to come in the form of institutional facilities in the focal LGAs either in schools or agreed public places. The suspension started on October 2017 and was lifted in March 2018 after a detailed investigation and evaluation of the project. UK Aid was satisfied with the findings from the evaluation done and therefore decided to lift the suspension. UP was asked to review its plan in the light of the delays and a no cost extension was approved for the project. United Purpose, the grant holder for the UK Aid Direct funding for the CHISHPIN project, demonstrated good value for money in its management of project implementation through scaling up of CLTS implementation from 514 communities (originally planned) to about 1180 CLTS 35

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report communities (by sub-division of too large communities to manageable CLTS communities) using the same funds. UP procurement processes and fund management were effective. UP has a comprehensive procurement policy and finance manual that guides its procurement procedure, instructions, and provides further guidance for carrying out procurement and financial activities for or on behalf of UP effectively and efficiently in compliance with its financial regulations and rules, and other applicable UP legislative instruments. As part of its implementation of the procurement policy, UP makes use of committee-based system because its procurements are not very large and frequent. Members of the committee are members of Country Management Team (CMT) made up of Country Director (CD), Finance Manager, Programme Manager, Office Manager plus Grant Accountant and State Technical Officer who are invited to be part of the process from time to time. UP has a list of suppliers determined through previous experience. Suppliers are also obtained from an INGO forum (where information about suppliers are shared). The INGO forum is a gathering of UK INGOs in Nigeria. They share best practices based on their experiences of working with reputable suppliers. UP adopts and makes use of a threshold to determine its suppliers. This involves; • •

Doing a verification and checking for compliance with its existing standards (including specific donor requirements).

Another key component of UPs VFM approached is how it measures its cost effectiveness. This involves; • • • • •

Reviewing costings through a market survey. Then carrying out analyses of costs. Finance team looks at the burn rate against their key budget drivers. They then focus on these drivers and negotiate the rates with service providers or suppliers. They ensure there is a good planning by reviewing the programme plans, and advice the programme team of duplication (where applicable) and then look at how to improve the logic to save money and ensure cost effectiveness.

There is no doubt that CHISHPIN project is a success from the point of view of the results it set out to achieve. There were drivers and barriers however that played significant roles in the outcomes and outputs achieved by the project. Key amongst the drivers is UP’s use of the traditional institution as part of the implementation of the CLTS process. The traditional rulers, because of the leadership position they occupy within the community, have been used as agents of change to facilitate the behaviour change experienced in the communities that led to ODF of whole communities and LGAs. Added to the use of the traditional institution, is the use of Natural Leaders (NL) as great facilitators of change. These are the so-called foot soldiers that emerge from within the community during the triggering process and become internal facilitators that drive the process of change. UP’s unique and smooth entry process has proved to be a key driver of its obtaining quick and effective buy-in amongst all the stakeholders in the CLTS process, beginning from the State level, through the LGAs to the communities. For instance, compared to other organizations in the same sector, UP has delivered proportionally more ODF LGAs using less funds and a shorter duration which proves both effectiveness and efficiency. Of course, UPs uniqueness is also due to its approach observed from the following; 36

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report • •

•

UP focuses on collective behavior change, NOT on providing infrastructure/hardware (toilets/latrines). UP approach is community based; involving the communities in all steps of the process; in case of difficulties/problems/refusal: the solution is always sought (and found!) within the community (e.g. by involving relevant/influential community members, “natural leaders”). UP is a small organization with short communication lines, it has built teams in the right sense of the word team. The unique adoption of the concept of shame and disgust

all through the CLTS process, right from the triggering stage all through the peer review process, WASH clinics and follow-up, has proved to be a strong driver. UP adopted the institutional triggering approached it learnt from Uganda and this has helped to sensitise the LGAs to see the importance and relevance of CLTS as a change process. Finally, the prompt release of funds by the UK Aid Direct meant that the processes were not disrupted. Reference was made to the policy of “use or lose it” where grant holders loss funds they could not use in the project to achieve desired results. The CHISHPIN project also experienced some barriers that affected the delivery of results. CLTS progress is hindered/slowed down during rainy season because of the bad roads especially in communities in Biase LGA. Another barrier experienced is the challenging areas to reach in some of the LGAs, again, especially in Biase where speed boats must be used before some communities could be reached. Communal conflicts made it impossible to have access to communities for follow-up. This was a common challenge in all the LGAs but more so in Biase communities where whole wards are affected. Lack of transparencies in some Civil Society Organizations (CSOs) engaged by the project. This led to some CSOs being disengaged from the project. Another barrier experienced is the indiscriminate transfer of trained LGA WASH department staff. These staffs are transferred, after being trained, to LGAs where there is no ongoing CLTS project. This creates skill gaps. Retraining takes time and so the project suffers. CHISHPIN project could be referred to as ‘a learning project.’ The project itself is a product of learning from the sister Rural Sanitation and Hygiene Promotion Programme in Nigeria (RUSHPIN) where CLTS is being implemented in three LGAs in Benue and three in Cross River State. CHISHPIN took the successful experiences from the RUSHPIN programme and adapted and applied them in the CHISHPIN project with outstanding results in a shorter time. The CHISHPIN project has consistently co-organized a quarterly review and planning meeting with the RUSHPIN project. These review meetings allowed a cross fertilization of knowledge, skills and field experiences from both projects which are then applied in the field to achieve results. Added to this is the WASH clinics organized by the CHISHPIN project that brings the project communities together to share progress and learnings from their projects. The communities that are not making progress learn and are challenged by the progress being made by their neighbouring communities and when they go back, they put into practice what they have learnt from the clinics. There has also been exchange visits that added value to the CHISHPIN project. These exchange support visits from successful LGAs like Obalinku (ODF in the RUSHPIN programme) provides hands-on training for CHISHPIN facilitators. CHISHPIN project has been involved in the annual national CLTS conferences where it had opportunity to share its learnings and learned from other practitioners. The project has however not been involved in sharing its learning internationally at conferences for example, partly because it did not have budgets for such events. The project has however been able to develop briefs and compendium of learning from its project which it has shared within the sector. 37

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report

3.5 Efficiency CHISHPIN project logframe has two milestones and a target for each outcome and output with clear timelines of expected delivery. It was therefore straight forward to correlate that the results were delivered within the planned delivery dates. A review of the CHISHPIN project logframe showed an initial delay in meeting milestone delivery dates in year one and a couple in year two. Annex VII shows a table of achievement against milestones 1, 2 and targets. Progress against each was determined from the annual reports prepared by UP and submitted to UK Aid Direct. That analysis showed that 16 of the 19 milestone 1 were not met at the time planned. Some of those not met were moved to later dates in year two and three. Most of the reasons proffered for missing these timelines is the uncompleted baseline. It was also observed that the project timeline for year 1 was very short (Nov 2015 to March 2016). Further analysis shows that the project was able to achieve planned results after the initial delays and, in some instances, even exceeded them. For example, the project planned to achieve “107 (40%) EHCs established and are conducting hygiene and health activities in schools and communities (Dec 2017).” At the time of reporting against this milestone, the project achieved 110 at the intermediate point and 125 EHCs established at the end of the project. In another instance the project set a target to achieve “268 (100%) WASHCOM established” by the end of the project. The final result shows 1052 WASHCOMs were established. Moreover, 12,591 WASHCOM members were trained as against 3,216 targeted. UP’s team adopted very practical approaches to dealing with cost drivers. This evaluation shows that both the project and finance teams have good understanding of cost drivers and used them to ensure VFM as explained in section 3.4 and below. From the programming point of view, UP recognised workshops and training as one of the cost drivers. As a result, it adopted a hands-on approach rather than organise workshops therefore saving cost. The project also decided to move meetings to ward levels where participants can travel from their various homes attend meetings and return same day without payments of accommodation and perdiems. The project adopted the approach of sourcing for vendors from the LGAs, rather than bringing them from Calabar. Payments to CSOs, which is an important cost driver was based on performance thereby reducing wastages and saving money for the project. The use of natural leaders to facilitate processes in other communities, instead of bringing outside facilitators, was another way that cost was saved. They also ensured use of competitive bidding process where they were able to engage the lowest bidders. From the finance point of view, effort was made to have a good understanding of the field programmes to identify the cost drivers for example training, travels, perdiems, hall rentals etc. Finance then concentrates on the key things that helps to deal with these cost items to ensure they are controlled, and best prices are obtained for them. After key cost drivers are identified, they engage with suppliers that can provide the service at much lower cost. For an example, they could engage a caterer that is cheaper than buying food from the hotel during events hosted in a hotel. In doing this, they ensure quality is not compromised. Another example is that they book tickets online themselves instead of using travel agencies and make good savings.

3.6 Sustainability CHISHPIN project has not leveraged any substantial funding from any sources. Government counterpart funding commitments were not met. CHISHPIN project benefitted from funding provided by PZ Cussons to support annual Global Handwashing events in which communities from the CHISHPIN project benefitted.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Going by the definition of ‘in-kind contributions’, which includes “professional services or expertise in the form of staff time.” (IFAD 2018, p. 1), the inputs in time and staff from WASH department staff, CSOs, WASHCOMs, Natural Leaders, and even Paramount Rulers based at LGA could be considered in-kind contributions. While, CHISHPIN project achieved great success and provided visible social, economic and health benefits to the poor and vulnerable in the LGAs, there is a grave concern that these benefits may not be sustained in the long run for the following reasons; • •

•

Lack of commitment and interest by the State Government to fund sanitation in the State Even though there is a semblance of budget for Environmental Health Department, there are no funds at the LGA and no release. In addition, there is no evidence of any budget for the WASH unit. There is no assurance there will be follow-up with communities by either LGA WASH unit, LTGS, STGS or CSOs because currently, there is no mechanism of funding in place to facilitate any of these groups when CHISHPIN funding ends FGD with WASHCOMs revealed that they may not be self-motivated to continue the work in the absence of external facilitation and follow up. Many members have left and there has been complaint of lack of payments during the project implementation. Absence of peer review process hitherto achieved during clinics means the driving force that challenges leaders to go back to their communities to effect changes is no longer available. Feedback received from WASH coordinator of Yala, during visit to communities to confirm some results, show that WASHCOM leadership is already being abandoned and some community members are already going back to old behaviours because they feel the project has ended.

In an interview with Yala LGA HOLGA, she said she realises that finance is very necessary, she however says finance is a challenge. According the HOLGA, to mobilise the people requires money. The LGA is not an autonomous organ, it depends on funding it receives from State Government. Asked if the EH department will be put on a budget, she responded that “there is already a budget but when the money does not come from the State Government then there is nothing they can do.” Discussion with the UP management reveal that there may be a chance of UP still working with the Yala and Ikom LGA if they are successful in a recent bid to UK Aid (2019).

3.7 Impact Two CSOs, Action for Rural Development (AFRUD) and Life Empowerment Foundation (LEF), were interviewed as part of this evaluation process. In response to the question “was your capacity built on this project? In which areas? Both organisations responded that their capacity was built in the following areas; •

CLTS Facilitation - CLTS triggering tools - How to conduct follow up – Madonna • Monitoring and Evaluation and Reporting • Accounting/Financial management • Trained on menstrual hygiene. 39

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report •

Training on community advocacy.

From the results achieved by the CSOs facilitating and following up communities to ODF while working alongside the LGA WASH departments in the three LGAs, it is very evident that the capacity building of the CSOs was very effective, because the CSOs were able to apply their training to facilitate communities to ODF. The influence of the CHISHPIN project was seen and felt beyond the boundaries of the project LGAs as evident in the case of the experience in Ikom LGA. A National Youth Service Corps (NYSC) member posted to Etung LGA (a neighbouring LGA to Ikom LGA) wrote to invite the Environmental Health Officers of Ikom LGA to come and provide a one-day sensitization training in the Effraya community of Etung LGA, where he was posted. In his request letter dated 18th February 2019, the NYSC member said “on my arrival to Effraya, I discover most houses do not have toilets and few that have, they are not properly use and clean. And it is hazardous and unhealthy for them.” Probably because he saw the result of the CLTS intervention in communities of Ikom, the NYSC member said, “I want to combat this indecent calamitous attitude in the community through Environmental Officers (WASH).” It is not as if Etung LGA does not have Environmental Health Department and probably a WASH unit, but because the CHISHPIN project is not in Etung LGA, they do not have the capacity to facilitate Effraya community to construct latrines and keep them clean, hence the reason for his request to Ikom LGA because of the results he has seen achieved through the support of the WASH unit of Ikom. At the end of the CHISHPIN project, 358,886 poor and marginalized community members in the three project LGAs have access to latrines and handwashing facilities and no longer defecate in the open. These include 202,542 who built basic latrines of their own. 755 disadvantaged people (elderly, widows, people living with HIV Aids) who now have their own latrines, 392 schools with adequate sanitation facilities, 24, 291 girls who have benefitted from having separate toilets. In addition, 1,052 WASHCOMs were established and trained resulting in 12,591 community WASHCOM members with leadership skills, skills in construction of latrines and Handwashing facilities. (see annex VII for achievement against milestones table). The medical records available shows that diarrheal incidence has reduced among the children under five, and the testimonies coming direct from the beneficiaries interviewed during the FGDs was overwhelming that there has been no cholera outbreak in the three LGAs since the CHISHPIN project started, All of these benefits testified to by the beneficiaries (see annex VIII) and those itemised above could not have been possible without UK Aid Direct funding through the CHISHPIN project.

4.0 Conclusions 4.1 Summary of achievements against evaluation questions The UK Aid funded CHISHPIN project turned out to be a relevant project in the context of the SDGs. It has contributed towards SDG 3 and 6 for CRS. It has created access to Sanitation for at least 358,886 poor and marginalised people of CRS. It has built capacity of CSOs to be able to facilitate CLTS in rural communities. In the words of the community beneficiaries of the project, it has saved them money they would have spent on hospital bills. They testify that they no longer experience cholera, diarrhoea and other sanitation related diseases. Open defecation has been completely eradicated in two and half LGAs. They now breathe in fresh air, they can now plant vegetables in their backyards and are now comfortable to pick fruits that fall from trees without the fear that it fell on excreta.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report The project demonstrated value for money through its grassroot programme implementation strategies that led to cost savings. UP, the grant holder, has in place a comprehensive policy that guides its procurement processes. Three years data (2016 – 2018) collected from the two health facilities in two communities and from the State Epidemiology Unit, though significant, is not sufficient to wholly, scientifically, attribute reduction in diarrhoea cases reported in under-five children in the focal LGAs to CHISHPIN project intervention. It however provides an indication of the intervention being a necessary triggering cause, because as Stern et al (2012) explains, it is an initiating contributory cause. CHISHPIN project intervention effect is classified as contributory because there are other interventions, identified as part of the causal package, earlier, that are ongoing within the intervention area that could also be contributing to reduction of reported diarrheal cases in the focal LGAs. Interventions like; exclusive breastfeeding for the first six months of life; health education about how infections spread; and rotavirus vaccination. These have been declared by WHO as contributing to reducing diarrheal cases in under-five children. Sustainability of the project after UK Aid funding stops is in doubt because there is no commitment from the State Government towards Sanitation development, The LGAs do not have funds dedicated to WASH implementation. Without follow up of communities by the LGA WASH department or CSOs, probability that the communities will relapse is high. The project did not leverage resources from other sources but enjoyed in-kind inputs from the LGAs and communities. The project delivered results despite delays in start-up, the five months suspension explained earlier, communal conflicts and difficult/challenging terrains notwithstanding. Without the funding from UK Aid Direct, 358,886 poor and marginalized people of CRS would still be without access to sanitation and would not be open defecation free environment; 1052 WASHCOMs would not have been established; 12,591 WASHCOM members including women will not be skilled in CLTS facilitation; capacity of 3 LGA WASH Departments and CSOs in the 3 focal LGAs would not have been built. The situation depicted in the four graphs (Figure 17, 18, 19, 20) can only mean one thing, that there may be other things happening in the communities (for example absence of clean water supply for drinking, poor administration of ORT ) to keep raising the diarrhoea levels despite the testimonies from the communities of the benefits they have received from the CHISHPIN CLTS project. This is not to say the CHISHPIN project has not made an impact but it simple signifies that there are other reasons responsible for the spikes in diarrhoea in under five children in the three LGAs. This evaluation cannot confirm what is responsible specifically but can only speculate considering the causal package identified for this evaluation. It is clear however that the CHISHPIN project did contribute to the reduction in diarrhoea cases in the under-five children within the project period.

4.2 Summary of achievements against rationale for UK Aid Direct funding One of UK Aid’s priorities is to tackle extreme poverty and help the world’s most vulnerable. One of the ways UK Aid intends to achieve its priorities is to improve access, supply and quality of basic services, in this case sanitation and hygiene. UK Aid Direct’s aim is to support Civil society to deliver solutions to achieve sustained poverty reduction and achieve the global goals. This evaluation believes UK Aid Direct’s aim and its priorities were achieved through its funding of UP as grant holder of the CHISHPIN project in the three LGAs of CRS. Poor and marginalized communities now have access to sanitation and hygiene with the economic, social educational benefits they go with. Girls now have access to toilets in school. They do not need to stay back 41

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report at home when they are in their menstrual cycle, in their own words they do not have to leave school halfway because they want to use the toilet. Women and girls do not need to expose themselves because they want to excrete, they can now comfortably use latrines in privacy and their dignity is preserved. The risk of snake bites in the bush, rainfall, harsh sun and that of being attacked because of going into the bush is now a thing of the past for at least 356,886 poor and marginalized community members. Socially they are proud to receive visitors to their homes with no concerns of having to take them to the bush in the night when they want to defecate. Above all, in the words of the people, “they no longer eat their own shit.” These, and many more testimonies (see annex VIII) from the communities are evidence of the benefits the communities have obtained from the funding and support provided by UK Aid Direct.

4.3 Overall impact and value for money of UK Aid Direct funded activities Health records at the community health centres visited was inadequate, but data obtained from the epidemiology Unit confirms the claim of community members that diarrhoea disease incidence has reduced in children. The testimonies from the communities about the reduction of diarrhoea and the absence cholera since the CHISHPIN project commenced is overwhelming and cannot be ignored. It is the communities’ experience and they are happy about it. The evaluation showed that the grant holder was cost-effective in its processes and understood the cost drivers in this project and managed them effectively and efficiently thereby demonstrating good value for money.

5.0 Lessons learnt (where relevant) 5.1 Project level - management, design, implementation The conventional approach to sanitation service delivery is usually mostly done together with water supply. This is however not so with the CHISPIN project. The ability of UP to market sanitation without the component of water supply in the programme is an innovative and laudable effort. It is therefore important to recognize that it is possible to achieve open defecation free status without water supply which is what they have demonstrated. Their approach utilizes traditional community institutions to drive CLTS implementation. Furthermore, CHISHPIN project has demonstrated that using sanitation intervention as an entry point is possible. With this approach, Sanitation will no longer trail Water Supply as has been the case where Sanitation is relegated to the back because communities embrace Water Supply to the detriment of Sanitation. Also, important as a strategy is embedding technical support (the LTSOs) within each LGA WASH unit. Their key role was ensuring quality control & compliance, hands-on mentoring and supervising delivery according to agreed work plans. This made a significant difference in the drive to facilitate ODF and to strengthen the capacity of the WASH units. Peer influence and the strategy of using disgust and shame helped communities review their sanitation status and take necessary action for improvement. This is approach is resulting in a self-sustaining strategy for getting communities to take ownership of the process with little support of the programme team. CHISHPIN project has further confirmed that adopting lessons from a project saves time and cost that would have been wasted re-inventing the wheel and making mistakes that could have been avoided. The success seen in the CHISHPIN project was largely due to the lessons learned from a sister project “RUSHPIN” managed by the same programme office.. What would have been pitfalls were already learned as lessons from the RUSHPIN experience, making CHISHPIN a model

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report A significant lesson from the CHISHPIN project centres around the use of population figures in projects. United Purpose obtained 2006 national census figures (702,000) and used that to design the CHISHPIN project and apply for project funding support. Verification done later by UP, through a peer review process involving house to house counts however showed that the census figures were too high. Unfortunately, baseline population count used an approach that involved extrapolation and at the end did not detect any issues with the initial figures except that it arrived at a much higher population figure of 774,354. As far as this evaluation result goes, UPâ&#x20AC;&#x2122;s population figures are closer to the reality on ground because it is very close to the figures obtained by the communities. This is a clear indication that population census figures cannot be trusted. Designing and planning a project with wrong information at the onset could lead to faulty project results.

5.2 UK Aid Direct management UK Aid Directâ&#x20AC;&#x2122;s policy of use it or loss it has been lauded by the grant holder field staff as very effective driver that has contributed to reducing laxity amongst project partners and make them focus on making good plans and focusing on them to achieve good results. Not having funds for sharing far and wide and policy influencing meant that the innovations, strategies and unique features of the CHISHPIN achieved at grassroot level remained there and did not have desired impact it should have at national and international arena.

6.0 Recommendations The evaluation found that there was no budget for sharing of learning nationally and internationally. While the CHISHPIN project had opportunity to share learning in the National CLTS conference, it could not organize national level learning events nor attend any international conferences that would have brought more attention to the successes it has achieved at the grassroots levels. It is recommended that in future adequate provisions be made not just for internal learning within the project, but for deliberate sharing of successes achieved at project level at national and international arena or events. Sustaining benefits of any intervention is very critical. In the light of the current situation with the CHISHPIN project at its closing, it is recommended that UP source for funding that will enable it further work with the LGAs to consolidate the benefits already achieved. Strategies that are focused on empowering the communities directly to be able to support their successes on their own should be developed and implemented. Considering the inconsistencies identified with population census figures, it is recommended that a thorough and factual approach be adopted at baseline to ensure that the correct population data of the project communities be obtained. Achieving this will require spending more funds and time for baseline surveys. This is important to avoid basing project design and planning on wrong population information at the onset of the project. It was observed that there was not a strong advocacy and influencing component to the CHISHPIN project. This may be a reason for the low advocacy and influencing specific activities particularly as it relates to supporting CSOs (in the area of policy influencing and advocacy) and the media capacity in this area. It is recommended that UK Aid Direct, because of its aim of supporting Civil society to deliver solutions to achieve sustained poverty reduction and achieve the global goals, emphasis and provide the necessary funds so grant holders of such interventions can include an elaborate advocacy and policy component which will ensure CSOs and media capacity is built in this area.

Community-led Health Improvement through Sanitation and Hygiene Promotion in Nigeria (CHISHPIN) Final Evaluation Report Based on the failure of the Government to meet its funding obligation for this project, it is recommended that Government not only be made to sign firm commitment agreements at the beginning of the project, but it should be made to put down funds upfront as a demonstration of its commitment to the implementation of the project.

REFERENCES Bamberger, M., Rao, V., and Woolcock, M. (2010). Using Mixed Methods in Monitoring and Evaluation. Experience from International Development. Policy Research Working Paper 5254. The World Bank Development Research Group. Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., and Befani, B. (2012). Broadening the Range Designs and Methods For Impact Evaluation. Report of Study Commissioned by Department for International Development. Working Paper 38