Evaluations: Value for Money Spotlight Brief, Sisters for Sisters’ Education Project, Nepal - 2021

Sisters for Sisters’ Education Project, Nepal

This Spotlight Brief is one of a series of Value for Money papers. It reviews the value for money (VfM) of five significant interventions from the Sister for Sisters’ Education project in Nepal implemented by VSO and funded by UK aid through the Girls’ Education Challenge between 2017 and 2021. It assesses intervention costs against the benefits delivered, by exploring the intervention’s relevancy, cost-effectiveness and sustainability. It is primarily aimed at GEC project partners, implementors and non-governmental organisations who are interested in assessing VfM and understanding which interventions provide good VfM in education programming.

1. Introduction

Funded by the UK Foreign, Commonwealth and Development Country Office (FCDO) through the Girls’ Education Challenge (GEC) and implemented by Voluntary Service Overseas (VSO), the Sisters for Sisters’ Education project worked with 49 schools to improve the education of 7,382 marginalised girls in Nepal. Running from 2017 and 2020, the project helped girls to transition from primary to secondary school and leave school ready to continue their education or secure sustainable employment. Other important outcomes included improved self-esteem and a greater sense of empowerment for the girls, and engagement with communities.

This brief reviews the VfM of five main project interventions:

1. Mentoring scheme for girls – 320 well-performing “Big Sisters” in higher grades provided academic support and mentoring to 1,200 highly marginalised “Little Sisters” in lower grades to help give them a voice and agency. The Big Sisters regularly engaged with parents and the school. The Little Sisters also received in-kind support such as stationery items, sanitation kits and uniforms. The support also included 147 adult champions (including teachers) who engaged in dialogue with the

community about girls’ education. The project also provided training and capacity building for the Big Sisters and champions and led awareness campaigns against harmful social norms.

2. EDGE (English and Digital Girl’s Education clubs) – Extra classes were provided in schools to introduce technology for digital learning and English. 1,392 girls were EDGE club members.

3. Learning Support Classes – Extra lessons were provided within the school for a subset of girls at school by paying the teachers for extra time after school for three months a year. The emphasis was to help the girls in their weak areas of learning.

4. Parental workshops – These workshops covered all 7,382 project girls. They addressed social norms about girls’ marriage and education by engaging with parents and raising awareness through dialogue, campaigns and street drama.

5. Teacher training – Training was provided to 1,220 teachers. It covered topics such as teaching methodology and strategies to promote inclusive education practices and development of low-cost teaching materials. It also covered child protection policies and mechanisms, psychosocial training, ICT training and creating a positive learning environment.

©

VSO, Sisters for Sisters'

THE GEC VFM FRAMEWORK

When reviewing VfM for GEC projects, the GEC VfM framework is applied systematically, drawing on evidence from the endline evaluation findings and project staff interviews. The VfM framework uses four of the OECD DAC criteria:

1. Relevance – whether the project invested in the right activities and modalities to respond to the needs and barriers of the girls identified, with optimal resources allocated to them.

2. (Cost)-effectiveness – whether the project produced the expected outcomes at an optimal cost for the girls and others reached by the project.

3. Efficiency – whether the project delivery was on time, on budget and with good quality processes.

4. Sustainability – whether there has been a long-term continuation of outcomes for the girls and others reached by the project and whether there has been replication and/ or scale-up or adoption without FCDO funding.

Read more about the GEC VfM framework and review methodology here

This VfM review draws on evidence from the project endline evaluation and interviews with the project team. The project endline evaluation consisted of a longitudinal study with a mixed method design for data collection for its second phase (2017 to 2021), with a mixed approach for data collection.

KEY FINDINGS FROM THE VFM REVIEW 1

• The project as a whole offered good value for money. It was cost-effective, delivering 0.25 years (3 months) of education per $100 spent for all girls supported.

• The mentoring scheme coupled with the Learning Support Classes had a fair VfM, being highly relevant and driving learning outcomes. Cost effectiveness could have been better with more girls per mentor targeted to capitalise on the high fixed costs of Big Sister training.

• Teacher training was not good VfM as it did not drive outcomes, partly due to its design and due to the COVID-19 school closures.

• The mentoring scheme for girls can potentially be replicated through local government at a lower unit cost given that fixed capacity building and training costs may not need to be repeated or could be spread across more girls.

• If the project were to continue it is worth remembering that parental awareness and streamlined mentoring for girls offered better value for money than the EDGE clubs and teacher training.

2. What were the project costs?

Figure 1 illustrates the percentage expenditure breakdown by interventions for the project’s entire duration. The greatest cost categories are the mentoring scheme for girls and EDGE clubs. The total cost per girl (of all girls covered within treatment schools) for all interventions combined is £586 or an annual cost of £147. This is comparable to the average expenditure per student for 2019/20 for secondary school in Nepal is £1552. It must be noted that this is not a ‘like for like’ comparison; Sisters for Sisters’ activities had a far wider focus than just education and learning. They covered girls’ empowerment, self-esteem and community engagement with girls’ education, arguably delivering more for a similar cost.

Figure 1: Intervention costs as a percentage of the total project budget (2017 to 2021)

BigSisterMentoringLearningSupportClassesLifeSkillsincludingASRHEDGEclubs Parental engeagementTeachersupportPolicyadvocacyLivelihoodsupport

Source: Sisters for Sisters’ project financial data

The livelihood and policy advocacy interventions comprised 13% of the budget and have not been considered in this brief due to difficulty in implementation as a result of COVID-19. The Big Sister network also provided policy advocacy work within the schools, ward and municipal levels. The fixed costs of monitoring and evaluation, central administration and COVID-19 extra costs (19.5% of the expenditure) have been fully allocated across the activities, pro rata. These are within normal expectations. The total expenditure includes a matched contribution from VSO, comprising 21% of the total budget.

1 Please note that the VfM review. The purpose of these reviews is not to question project aims and objectives. Instead, it is to examine the evidence from a project to judge the degree to which interventions provided value for money. The reviews also recognise that projects operate in different contexts, which means the findings are not generalisable. Furthermore, VfM is one dimension, among many, by which the relative successes of programming can be judged.

2 Source: Peano 2020 EMIS database. Ministry of Education Science and Technology. ASIP & AWPB (CEHRD 2019c) cited in Peano 2020.

3. Which interventions were the most relevant?

The mentoring scheme for girls was highly relevant. It targeted the most marginalised girls by addressing girls’ lack of voice, agency and confidence. It did this in a sharply focused and highly appropriate way. However, it comprised the greatest proportion of the expenditure – almost a third. The remuneration (on a par with volunteer teachers and facilitators) given to the mentors and adult champions was not driving this cost, it was the capacitybuilding activities for the Big Sisters. This was necessary due to the increasing complexity in mentoring as the girls got older. The training was highly relevant in addressing mental health issues in a child-centred school and community culture and filled a gap.

The Learning Support Classes were also highly relevant as students in Nepal are often found to be lacking in English, maths and Nepali – a focus of the Learning Support Classes. Evaluation findings indicated that the Learning Support Classes directly addressed this barrier faced by many girls. The Learning Support Classes intervention comprised only 4% of the budget as this cost only consisted of teacher remuneration.

The EDGE clubs’ component showed some relevance, as findings from the endline evaluation indicated that this filled the gap of a lack of computers in schools and knowledge on how to use them. The technology enabled a more interactive environment within the English classes which is often lacking and results in students failing English exams. Education technology interventions can be expensive by nature. Thus, a high amount of resources had been allocated to this, comprising roughly 23% of the budget. The relatively high cost could perhaps be explained by the use of use of laptops rather than smartphones, which are cheaper and more easily accessible in Nepal. Staff interviews indicated that retaining girls in EDGE was also a challenge and that attendance was always an issue. However, this was resolved by expanding access to less marginalised girls.

The parental workshops were also relevant as the endline evaluation found that they addressed the barriers reinforced by parents on girls, including chore duties and low parental engagement regarding education. In general, such deep-rooted barriers need to be urgently addressed by relevant interventions in order to create sustained behaviour change. Resources allocated to this intervention comprised 6% of the budget. This relatively low-cost allocation potentially explains its lack of effectiveness (discussed below). Perhaps more investment in this important area of unmet needs could have improved the effectiveness and overall VfM of the project.

Teachers training was found to have the lowest relevance mainly due to its content, modality and teachers’ limited capacity to absorb it. The endline evaluation found that the interactive teaching style was found to be difficult to implement in a context where traditional ways of teaching and corporal punishment were widely used. And a high burden on teachers generally meant that finishing the course on time within limited class hours was a challenge for them. It was also difficult to motivate government employed teachers, who were not directly employed by and accountable to the project. In terms of the modality there was an over-reliance on relatively expensive consultants and one-off residential training events for a limited number of participants in the first half of the project rather than implementing the proposed continued professional development approach. Taking up 16% of the total project budget (mostly through matched funding from VSO), the amount spent on this could potentially have been better spent on other things.

4.

Which interventions were

the most costeffective?

Project learning gains are estimated to be equivalent to an extra 0.25 years (3 months) of additional education per $100 spent. This estimate is based on two main data points from the total expenditure and midline evaluation:

1. An average cost of £576 per girl for the entire project duration, and

2. The control group learning gains in formal education in a year.

Endline data was not available due to the COVID-19 school closures. This result most likely has the effect of understating costeffectiveness. It is plausible that the learning outcomes as measured at endline could have been greater within the same budget envelope if the pandemic had not happened. In addition, the project achieved much more than just learning gains, further understating this result. Even so, this result is on a par with other similar education interventions (Figure 2 below) thus suggesting good VfM overall.

Ironand vitaminAIndia Villageschools Afghanistan FellowshipschoolsPakistan

SistersforSistersNepalSecondaryscholarshipsKenya

The Sisters for Sisters’ interventions addressed myriad barriers, with varying degrees of effectiveness. The greatest effectiveness and efficiencies were achieved by the flagship mentoring scheme as demonstrated in the endline evaluation. At endline both parents and girls stated that the mentoring scheme was one of the most significant interventions. The endline evaluation found that the Big Sisters enabled the Little Sisters to have confidence, voice and agency by raising problems with teachers, addressing absenteeism, mediating between the parents and the schools and enabling learning. The mentoring scheme had the dual objective of improving learning outcomes and also developing life skills. However, the relatively high cost per girl of £1,152 for the four-year duration of the project reflected the relatively low reach of mentoring and corresponding low workload of the Big Sisters, compared to similar community champions. Cost effectiveness could have been higher if the reach had been higher, as the fixed costs of training could have been spread over more girls, thus reaping economies of scale.

The Learning Support Classes were also found to be highly effective and efficient by the endline evaluation. Girls highlighted that these helped them improve their ‘examination score’,

Figure 2: Example cost-effectiveness ratios from other education projects worldwide

Source: Taken from Jameel Poverty Action Lab (2017) from United Nations Girls’ Education Initiative and Malala Fund, ‘Spending Better for Gender Equality in Education: Why the quality of financing matters for girls’ education, and what to do about it’, New York, 2021

‘interest to learn’, ‘classroom participation, and ‘comprehension of the lesson’. These child centric classes equipped the girls with confidence to question the teachers free from judgement. Initially the Learning Support Class was for weaker students in the class, giving the girls ample opportunity to re-learn the lesson from the basics repeatedly without having to worry about being discriminated against. Given that they only comprised 4% of the budget as a whole, this was a very cost-effective activity. Evaluation findings at endline suggest that the Learning Support Classes together with the mentoring scheme drove the outcomes. Together these activities comprised 36% of the budget. Based on 1,200 Little Sisters benefiting directly from the mentoring and adding in the Learning Support Classes costs, this gives rise to a cost per girl of £1,280 for the four-year duration of the project.

The endline evaluation found that the EDGE clubs also demonstrated some good value in terms of effectiveness and efficiency by providing the girls with an unprecedented opportunity to be acquainted with digital skills for the first time, and allowing them to learn English in an interactive environment, thus closing the digital and English learning gap. The girls attributed learning success to the EDGE clubs which boosted their confidence in using computers and also increasing their confidence in English. EDGE clubs were implemented for two years, the rest of the project time was used for preparation. Project staff indicated that the effectiveness was compromised by a limited reach, and limited skill of the facilitators. Based on 1,392 girls directly benefiting from EDGE clubs, this gives rise to a cost per girl of £728 for the project entirety.

The endline evaluation found that the teacher training component was reasonably successful in introducing new teaching methods based on greater empathy and reducing the number of cases of corporal punishment, but the design fell short in terms of the feasibility to adopt child-friendly teaching methodology in the classrooms. This activity covered all the girls. The low effectiveness with a sizeable cost allocation to this activity (16%) with a cost per teacher of £581 for the project duration for 1,220 teachers meant that this component slightly undermined overall VfM of the project. It must be noted, however, that this component was compromised by the school closures of the COVID-19 pandemic as VSO volunteers had left and training could not go ahead.

In addition, the use of international volunteers may have compromised efficiency and effectiveness, by creating delays in their deployment. Project staff interviews indicated that by midline just two international volunteer teacher trainers had been recruited. The priority focus of the training design was on residential events rather than ongoing mentoring which was not cost-effective. It was difficult to fill national staff posts due to greater remuneration for consultants. Overall, the lack of effectiveness can be partly attributed to factors out of control of the project but also largely to inefficient processes and programme management.

Similarly, the endline evaluation found that parental workshops had mixed results regarding behaviour change and effectiveness. Qualitative evidence collected as part of the endline evaluation suggested positive changes in attitudes towards girls’ education. However, there was only limited evidence amongst mothers actively inquiring about the progress of the girl and virtually no behaviour change in fathers. Unlike EDGE clubs, the mentoring scheme for girls and Learning Support Classes, parental workshops reached all the girls in the project population.

5. Which interventions were the most sustainable?

Despite some notable examples – such as the materials from the EDGE clubs being uploaded onto government systems –the project did not deliver much by way of sustainability, as measured by continued funding, replication or scale up within government.

The mentoring scheme for girls offers the most promising sustainability. Since the endline evaluation, there have been positive developments in terms of a current drive through the Sisters for Education Network, to achieve government buy-in and local replication of the Sisters for Sisters’ mentoring model. Also during the interviews the project team indicated that Big Sisters have been accessing opportunities for employment. Given that the bulk of the costs were fixed– training and capacity building – replication costs for the mentoring scheme could be significantly lower than the project £1,280 per girl in the future, as training and capacity building need not be repeated to such an intense degree. Indeed, the remuneration costs would arguably be the most important component in the future, and these were roughly only 2% of the total mentoring package costs. But they would be needed for scale up. Encouragingly, the Big Sisters stated in qualitative endline evidence that they would be very happy to continue their role if provided with some minimum form of remuneration. Parental workshops demonstrated some limited sustainability as parents’ increased awareness was likely to continue in the future by engaging with their daughters’ education beyond project closure.

The EDGE component does not offer sustainability due to the relatively high cost per girl of £728, thus preventing the ability of schools or local governments to adopt this going forward in the absence of donor funding for replication and scale up.

The Learning Support Classes also showed low sustainability due to uncertainty regarding remunerating the teachers by the schools or local governments, despite the relatively low cost. Most of the local governments and schools are already reeling under the pressure of low resources and in this light, it is unclear whether they will continue the intervention. The current dialogue between the project team and local government may lead to secure this continuation.

In terms of teacher training, there is limited evidence of changes in the behaviours of teachers. The endline evaluation found that teachers were now using more respectable language towards students. This can be claimed as another breakthrough of the project as teachers had become sensitive towards the dignity of the student and attempt to uphold that too. However, counter to this, teacher recruitment and deployment policy is shifting under a new federal system and this could undermine gains that have been made.

6. Conclusions

Findings from the VfM review suggests that the project showed good VfM. To illustrate the VfM of the specific interventions, Figure 3 plots the intervention costs against the benefits delivered, as proxied by effectiveness, sustainability and relevance. The top left quadrant, where a high value of benefits

was delivered through a low cost investment, has the highest VfM. Mentoring coupled with the Learning Support Classes offered the best VfM and falls into the ‘fair to good VfM’ quadrant. Teacher training has the lowest VfM.

The benefits are further broken down and loosely rated against each of VfM review criteria in Table

Figure 3: Investment costs versus benefits of each intervention

1.

Table 1: VfM ratings for interventions against VfM review criteria

Annex 1: Cost-Effectiveness Ratio (CER) estimation

Table 2 provides a baseline-midline comparison of Secondary Grade Reading Assessment (SeGRA) score, disaggregated by grade. Table 3 provides the literacy CER ratio estimation.

Table 4 gives the average Secondary Grade Mathematical Assessment (SeGMA) score, disagregated by grade. Table 5 provides the numeracy CER ratio estimation.

Table 2: Baseline-midline comparison of Secondary Grade Reading Assessment (SeGRA) scores

Source: Sisters for Sisters’ Education Midline Evaluation Report, 2019.

Table 3: Literacy CER ratio estimation

* statistically significant average difference in baseline score and midline score

Source: Sisters for Sisters’ Education Midline Evaluation Report, 2019

Table

Table 4: Average Secondary Grade Mathematical Assessment (SeGMA) scores

5: Numeracy CER ratio estimation

Annex 2: GEC VfM review methodology

The aim of the GEC VfM framework is to offer a quick pragmatic methodology to review the VfM of a GEC project by using existing evaluation findings. The framework uses the OECD DAC criteria (relevance, efficiency, effectiveness and sustainability) and evaluation findings, and reframes them through a VfM “lens” drawing out the key features of the findings that point to strong efficient value generation for the right people against optimal costs and resource allocation.

Effectiveness

The GEC approach to VfM analysis relies on the extraction of effectiveness data from the evaluation reports (midline and endline). This should include all the different types of outcomes assessed (learning, transition and sustainability) and the intermediate outcomes, such as wellbeing and life skills, self-esteem, and social norms and behaviour changes. It should also include data on how effective interventions have been for different targeted subgroups. All types of data used to demonstrate effectiveness are relevant for VfM purposes (quantitative or qualitative data, including the beneficiaries’ voices on what they found most valuable).

Effectiveness can either be assessed for a GEC project as a whole, or for separate components. Some projects’ midlines or endlines may be able to disentangle the impact and causality of certain interventions on outcomes over and above others. This likely will only be feasible for evaluations with a comparison group.

Cost-effectiveness

With activity-based budgeting, specific interventions can be assessed on cost-effectiveness. Costs can be presented in cost per girl format, with narrative attached to it, explaining what the overall cost per girl achieved in terms of outcomes observed. The number of girls reached by interventions can differ, thus giving rise to very wide-ranging figures. These variances should be discussed within the VfM analysis. Benchmarking to similar projects within the same context would be useful.

If there are strong, statistically significant findings, with a control group of girls displaying the counterfactual ‘without project’ learning achieved in a year of schooling, the analysis can be taken further to estimate the Cost-Effectiveness Ratio (CER): additional years of schooling per $100 spent.

Relevance

The GEC approach to VfM and relevance is to use evaluation report findings around relevance to understand whether resources were allocated to meet the needs and objectives identified upfront, and whether optimal resources were allocated across activities. If such needs changed over time, the VfM review should consider whether resources were reallocated to reflect this to maintain relevance of the programme.

An equity angle to relevance would determine whether or not the right beneficiaries were targeted by the project according to needs, and if enough or optimal resources were allocated to various targeted groups. The analysis should consider whether, in retrospect, the budget would have been carved differently across activities to reflect relevance better, and whether there was a correct allocation of funds for technical functions, monitoring and evaluation, and management etc.

Sustainability

Sustainability within the GEC is measured by:

1. A long-term continuation of outcomes for the direct beneficiaries themselves (including targeted schools and communities) and

2. Replication and/or scale-up or adoption of project activities without the need for FCDO funding.

A project may have sound input costs (Economy), have a demonstrated ability to translate its activities into quality outputs (Efficiency), and achieve its targets with regard to learning and attendance (Effectiveness), but may not have a strong sustainability case. Sustainability is not always covered in the conventional measures of VfM. But it is another factor to justify expenditure. The evidence of evaluation findings on contributions to sustainability should be integrated into a VfM narrative. Evidence of replication or scale up beyond project funding would point to very strong VfM. Sustainability intent may have been present from the start in the form of specific design features or plans. But over time, as contexts have changed, contributions to sustainability may not have materialised. This may require projects to undertake additional activities targeting sustainability that increase their costs but do not necessarily improve their efficiency or effectiveness in the short term. These should be considered in a VfM assessment. Another angle to considering costs and sustainability is defining the minimum spend for activities required to achieve sustainable outcomes. For example, determining the cost of a minimum amount of project exposure/duration or intensity necessary to achieve sustained outcomes based on findings.

Efficiency

Taking a narrative approach, assessing efficiency involves understanding how smoothly processes and interventions have been delivered (speed, quality, cost). There are four aspects to efficiency:

1. Whether the project as a whole was delivered on time and on budget

2. Assessment of the speed, quality and cost of the operating models for each intervention and the project as a whole

3. Assessing the efficiency of processes and management of the project as a whole

4. Assessing the efficiency of targeting girls (inclusion or exclusion errors).

Limitations of the GEC VfM approach

The VfM analysis is dependent on the evaluation findings, so its efficacy depends on the efficacy of the evaluation findings. It is also dependent on the ability of projects to producing expenditure data in relevant formats (activity-based budgeting). VfM assessments such as these are not often appropriate for making comparisons with other projects, due to differing contexts, cost structures and activities.

For more information, contact: learningteam@girlseducationchallenge.org | www.girlseducationchallenge.org

The Girls’ Education Challenge is a project funded by the UK’s Foreign, Commonwealth and Development Office (“FCDO”), formerly the Department for International Development (“DFID”), and is led and administered by PricewaterhouseCoopers LLP and Mott MacDonald (trading as Cambridge Education), working with organisations including Nathan Associates London Ltd. and Social Development Direct Ltd. This publication has been prepared for general guidance on matters of interest only and does not constitute professional advice. You should not act upon the information contained in this publication without obtaining specific professional advice. No representation or warranty (express or implied) is given as to the accuracy or completeness of the information contained in this publication, and, to the extent permitted by law, PricewaterhouseCoopers LLP and the other entities managing the Girls’ Education Challenge (as listed above) do not accept or assume any liability, responsibility or duty of care for any consequences of you or anyone else acting, or refraining to act, in reliance on the information contained in this publication or for any decision based on it.