BigSisterMentoringLearningSupportClassesLifeSkillsincludingASRHEDGEclubs Parental engeagementTeachersupportPolicyadvocacyLivelihoodsupport

Source: Sisters for Sisters’ project financial data

The livelihood and policy advocacy interventions comprised 13% of the budget and have not been considered in this brief due to difficulty in implementation as a result of COVID-19. The Big Sister network also provided policy advocacy work within the schools, ward and municipal levels. The fixed costs of monitoring and evaluation, central administration and COVID-19 extra costs (19.5% of the expenditure) have been fully allocated across the activities, pro rata. These are within normal expectations. The total expenditure includes a matched contribution from VSO, comprising 21% of the total budget.

1 Please note that the VfM review. The purpose of these reviews is not to question project aims and objectives. Instead, it is to examine the evidence from a project to judge the degree to which interventions provided value for money. The reviews also recognise that projects operate in different contexts, which means the findings are not generalisable. Furthermore, VfM is one dimension, among many, by which the relative successes of programming can be judged.

2 Source: Peano 2020 EMIS database. Ministry of Education Science and Technology. ASIP & AWPB (CEHRD 2019c) cited in Peano 2020.

3. Which interventions were the most relevant?

The mentoring scheme for girls was highly relevant. It targeted the most marginalised girls by addressing girls’ lack of voice, agency and confidence. It did this in a sharply focused and highly appropriate way. However, it comprised the greatest proportion of the expenditure – almost a third. The remuneration (on a par with volunteer teachers and facilitators) given to the mentors and adult champions was not driving this cost, it was the capacitybuilding activities for the Big Sisters. This was necessary due to the increasing complexity in mentoring as the girls got older. The training was highly relevant in addressing mental health issues in a child-centred school and community culture and filled a gap.

The Learning Support Classes were also highly relevant as students in Nepal are often found to be lacking in English, maths and Nepali – a focus of the Learning Support Classes. Evaluation findings indicated that the Learning Support Classes directly addressed this barrier faced by many girls. The Learning Support Classes intervention comprised only 4% of the budget as this cost only consisted of teacher remuneration.

The EDGE clubs’ component showed some relevance, as findings from the endline evaluation indicated that this filled the gap of a lack of computers in schools and knowledge on how to use them. The technology enabled a more interactive environment within the English classes which is often lacking and results in students failing English exams. Education technology interventions can be expensive by nature. Thus, a high amount of resources had been allocated to this, comprising roughly 23% of the budget. The relatively high cost could perhaps be explained by the use of use of laptops rather than smartphones, which are cheaper and more easily accessible in Nepal. Staff interviews indicated that retaining girls in EDGE was also a challenge and that attendance was always an issue. However, this was resolved by expanding access to less marginalised girls.

The parental workshops were also relevant as the endline evaluation found that they addressed the barriers reinforced by parents on girls, including chore duties and low parental engagement regarding education. In general, such deep-rooted barriers need to be urgently addressed by relevant interventions in order to create sustained behaviour change. Resources allocated to this intervention comprised 6% of the budget. This relatively low-cost allocation potentially explains its lack of effectiveness (discussed below). Perhaps more investment in this important area of unmet needs could have improved the effectiveness and overall VfM of the project.

Teachers training was found to have the lowest relevance mainly due to its content, modality and teachers’ limited capacity to absorb it. The endline evaluation found that the interactive teaching style was found to be difficult to implement in a context where traditional ways of teaching and corporal punishment were widely used. And a high burden on teachers generally meant that finishing the course on time within limited class hours was a challenge for them. It was also difficult to motivate government employed teachers, who were not directly employed by and accountable to the project. In terms of the modality there was an over-reliance on relatively expensive consultants and one-off residential training events for a limited number of participants in the first half of the project rather than implementing the proposed continued professional development approach. Taking up 16% of the total project budget (mostly through matched funding from VSO), the amount spent on this could potentially have been better spent on other things.

4.

Which interventions were

the most costeffective?

Project learning gains are estimated to be equivalent to an extra 0.25 years (3 months) of additional education per $100 spent. This estimate is based on two main data points from the total expenditure and midline evaluation:

1. An average cost of £576 per girl for the entire project duration, and

2. The control group learning gains in formal education in a year.

Endline data was not available due to the COVID-19 school closures. This result most likely has the effect of understating costeffectiveness. It is plausible that the learning outcomes as measured at endline could have been greater within the same budget envelope if the pandemic had not happened. In addition, the project achieved much more than just learning gains, further understating this result. Even so, this result is on a par with other similar education interventions (Figure 2 below) thus suggesting good VfM overall.

Ironand vitaminAIndia Villageschools Afghanistan FellowshipschoolsPakistan

SistersforSistersNepalSecondaryscholarshipsKenya

The Sisters for Sisters’ interventions addressed myriad barriers, with varying degrees of effectiveness. The greatest effectiveness and efficiencies were achieved by the flagship mentoring scheme as demonstrated in the endline evaluation. At endline both parents and girls stated that the mentoring scheme was one of the most significant interventions. The endline evaluation found that the Big Sisters enabled the Little Sisters to have confidence, voice and agency by raising problems with teachers, addressing absenteeism, mediating between the parents and the schools and enabling learning. The mentoring scheme had the dual objective of improving learning outcomes and also developing life skills. However, the relatively high cost per girl of £1,152 for the four-year duration of the project reflected the relatively low reach of mentoring and corresponding low workload of the Big Sisters, compared to similar community champions. Cost effectiveness could have been higher if the reach had been higher, as the fixed costs of training could have been spread over more girls, thus reaping economies of scale.

The Learning Support Classes were also found to be highly effective and efficient by the endline evaluation. Girls highlighted that these helped them improve their ‘examination score’,

‘interest to learn’, ‘classroom participation, and ‘comprehension of the lesson’. These child centric classes equipped the girls with confidence to question the teachers free from judgement. Initially the Learning Support Class was for weaker students in the class, giving the girls ample opportunity to re-learn the lesson from the basics repeatedly without having to worry about being discriminated against. Given that they only comprised 4% of the budget as a whole, this was a very cost-effective activity. Evaluation findings at endline suggest that the Learning Support Classes together with the mentoring scheme drove the outcomes. Together these activities comprised 36% of the budget. Based on 1,200 Little Sisters benefiting directly from the mentoring and adding in the Learning Support Classes costs, this gives rise to a cost per girl of £1,280 for the four-year duration of the project.

The endline evaluation found that the EDGE clubs also demonstrated some good value in terms of effectiveness and efficiency by providing the girls with an unprecedented opportunity to be acquainted with digital skills for the first time, and allowing them to learn English in an interactive environment, thus closing the digital and English learning gap. The girls attributed learning success to the EDGE clubs which boosted their confidence in using computers and also increasing their confidence in English. EDGE clubs were implemented for two years, the rest of the project time was used for preparation. Project staff indicated that the effectiveness was compromised by a limited reach, and limited skill of the facilitators. Based on 1,392 girls directly benefiting from EDGE clubs, this gives rise to a cost per girl of £728 for the project entirety.

The endline evaluation found that the teacher training component was reasonably successful in introducing new teaching methods based on greater empathy and reducing the number of cases of corporal punishment, but the design fell short in terms of the feasibility to adopt child-friendly teaching methodology in the classrooms. This activity covered all the girls. The low effectiveness with a sizeable cost allocation to this activity (16%) with a cost per teacher of £581 for the project duration for 1,220 teachers meant that this component slightly undermined overall VfM of the project. It must be noted, however, that this component was compromised by the school closures of the COVID-19 pandemic as VSO volunteers had left and training could not go ahead.

In addition, the use of international volunteers may have compromised efficiency and effectiveness, by creating delays in their deployment. Project staff interviews indicated that by midline just two international volunteer teacher trainers had been recruited. The priority focus of the training design was on residential events rather than ongoing mentoring which was not cost-effective. It was difficult to fill national staff posts due to greater remuneration for consultants. Overall, the lack of effectiveness can be partly attributed to factors out of control of the project but also largely to inefficient processes and programme management.

Similarly, the endline evaluation found that parental workshops had mixed results regarding behaviour change and effectiveness. Qualitative evidence collected as part of the endline evaluation suggested positive changes in attitudes towards girls’ education. However, there was only limited evidence amongst mothers actively inquiring about the progress of the girl and virtually no behaviour change in fathers. Unlike EDGE clubs, the mentoring scheme for girls and Learning Support Classes, parental workshops reached all the girls in the project population.

5. Which interventions were the most sustainable?

Despite some notable examples – such as the materials from the EDGE clubs being uploaded onto government systems –the project did not deliver much by way of sustainability, as measured by continued funding, replication or scale up within government.

The mentoring scheme for girls offers the most promising sustainability. Since the endline evaluation, there have been positive developments in terms of a current drive through the Sisters for Education Network, to achieve government buy-in and local replication of the Sisters for Sisters’ mentoring model. Also during the interviews the project team indicated that Big Sisters have been accessing opportunities for employment. Given that the bulk of the costs were fixed– training and capacity building – replication costs for the mentoring scheme could be significantly lower than the project £1,280 per girl in the future, as training and capacity building need not be repeated to such an intense degree. Indeed, the remuneration costs would arguably be the most important component in the future, and these were roughly only 2% of the total mentoring package costs. But they would be needed for scale up. Encouragingly, the Big Sisters stated in qualitative endline evidence that they would be very happy to continue their role if provided with some minimum form of remuneration. Parental workshops demonstrated some limited sustainability as parents’ increased awareness was likely to continue in the future by engaging with their daughters’ education beyond project closure.

The EDGE component does not offer sustainability due to the relatively high cost per girl of £728, thus preventing the ability of schools or local governments to adopt this going forward in the absence of donor funding for replication and scale up.

The Learning Support Classes also showed low sustainability due to uncertainty regarding remunerating the teachers by the schools or local governments, despite the relatively low cost. Most of the local governments and schools are already reeling under the pressure of low resources and in this light, it is unclear whether they will continue the intervention. The current dialogue between the project team and local government may lead to secure this continuation.

In terms of teacher training, there is limited evidence of changes in the behaviours of teachers. The endline evaluation found that teachers were now using more respectable language towards students. This can be claimed as another breakthrough of the project as teachers had become sensitive towards the dignity of the student and attempt to uphold that too. However, counter to this, teacher recruitment and deployment policy is shifting under a new federal system and this could undermine gains that have been made.

6. Conclusions

Findings from the VfM review suggests that the project showed good VfM. To illustrate the VfM of the specific interventions, Figure 3 plots the intervention costs against the benefits delivered, as proxied by effectiveness, sustainability and relevance. The top left quadrant, where a high value of benefits was delivered through a low cost investment, has the highest VfM. Mentoring coupled with the Learning Support Classes offered the best VfM and falls into the ‘fair to good VfM’ quadrant. Teacher training has the lowest VfM.

The benefits are further broken down and loosely rated against each of VfM review criteria in Table

Annex 1: Cost-Effectiveness Ratio (CER) estimation

Table 2 provides a baseline-midline comparison of Secondary Grade Reading Assessment (SeGRA) score, disaggregated by grade. Table 3 provides the literacy CER ratio estimation.

Table 4 gives the average Secondary Grade Mathematical Assessment (SeGMA) score, disagregated by grade. Table 5 provides the numeracy CER ratio estimation.

* statistically significant average difference in baseline score and midline score

Source: Sisters for Sisters’ Education Midline Evaluation Report, 2019

Table

Annex 2: GEC VfM review methodology

The aim of the GEC VfM framework is to offer a quick pragmatic methodology to review the VfM of a GEC project by using existing evaluation findings. The framework uses the OECD DAC criteria (relevance, efficiency, effectiveness and sustainability) and evaluation findings, and reframes them through a VfM “lens” drawing out the key features of the findings that point to strong efficient value generation for the right people against optimal costs and resource allocation.

Effectiveness

The GEC approach to VfM analysis relies on the extraction of effectiveness data from the evaluation reports (midline and endline). This should include all the different types of outcomes assessed (learning, transition and sustainability) and the intermediate outcomes, such as wellbeing and life skills, self-esteem, and social norms and behaviour changes. It should also include data on how effective interventions have been for different targeted subgroups. All types of data used to demonstrate effectiveness are relevant for VfM purposes (quantitative or qualitative data, including the beneficiaries’ voices on what they found most valuable).

Effectiveness can either be assessed for a GEC project as a whole, or for separate components. Some projects’ midlines or endlines may be able to disentangle the impact and causality of certain interventions on outcomes over and above others. This likely will only be feasible for evaluations with a comparison group.

Cost-effectiveness

With activity-based budgeting, specific interventions can be assessed on cost-effectiveness. Costs can be presented in cost per girl format, with narrative attached to it, explaining what the overall cost per girl achieved in terms of outcomes observed. The number of girls reached by interventions can differ, thus giving rise to very wide-ranging figures. These variances should be discussed within the VfM analysis. Benchmarking to similar projects within the same context would be useful.

If there are strong, statistically significant findings, with a control group of girls displaying the counterfactual ‘without project’ learning achieved in a year of schooling, the analysis can be taken further to estimate the Cost-Effectiveness Ratio (CER): additional years of schooling per $100 spent.

Relevance

The GEC approach to VfM and relevance is to use evaluation report findings around relevance to understand whether resources were allocated to meet the needs and objectives identified upfront, and whether optimal resources were allocated across activities. If such needs changed over time, the VfM review should consider whether resources were reallocated to reflect this to maintain relevance of the programme.

An equity angle to relevance would determine whether or not the right beneficiaries were targeted by the project according to needs, and if enough or optimal resources were allocated to various targeted groups. The analysis should consider whether, in retrospect, the budget would have been carved differently across activities to reflect relevance better, and whether there was a correct allocation of funds for technical functions, monitoring and evaluation, and management etc.

Sustainability

Sustainability within the GEC is measured by:

1. A long-term continuation of outcomes for the direct beneficiaries themselves (including targeted schools and communities) and

2. Replication and/or scale-up or adoption of project activities without the need for FCDO funding.

A project may have sound input costs (Economy), have a demonstrated ability to translate its activities into quality outputs (Efficiency), and achieve its targets with regard to learning and attendance (Effectiveness), but may not have a strong sustainability case. Sustainability is not always covered in the conventional measures of VfM. But it is another factor to justify expenditure. The evidence of evaluation findings on contributions to sustainability should be integrated into a VfM narrative. Evidence of replication or scale up beyond project funding would point to very strong VfM. Sustainability intent may have been present from the start in the form of specific design features or plans. But over time, as contexts have changed, contributions to sustainability may not have materialised. This may require projects to undertake additional activities targeting sustainability that increase their costs but do not necessarily improve their efficiency or effectiveness in the short term. These should be considered in a VfM assessment. Another angle to considering costs and sustainability is defining the minimum spend for activities required to achieve sustainable outcomes. For example, determining the cost of a minimum amount of project exposure/duration or intensity necessary to achieve sustained outcomes based on findings.

Efficiency

Taking a narrative approach, assessing efficiency involves understanding how smoothly processes and interventions have been delivered (speed, quality, cost). There are four aspects to efficiency:

1. Whether the project as a whole was delivered on time and on budget

2. Assessment of the speed, quality and cost of the operating models for each intervention and the project as a whole

3. Assessing the efficiency of processes and management of the project as a whole

4. Assessing the efficiency of targeting girls (inclusion or exclusion errors).

Limitations of the GEC VfM approach

The VfM analysis is dependent on the evaluation findings, so its efficacy depends on the efficacy of the evaluation findings. It is also dependent on the ability of projects to producing expenditure data in relevant formats (activity-based budgeting). VfM assessments such as these are not often appropriate for making comparisons with other projects, due to differing contexts, cost structures and activities.