Value for Money Spotlight Brief, Aarambha Project, Nepal - 2021

Aarambha Project, Nepal

This Spotlight Brief is one of papers looking at Value for Money (VfM) of education interventions. It reviews the emergent findings in VfM achieved for the interventions of the Aarambha project. This project is being implemented in Nepal by People in Need (PIN) and is funded by UK Aid through the Girls’ Education Challenge (GEC). Based on its Year 1 endline evaluation, it assesses intervention costs against the benefits delivered by exploring the relevance, cost-effectiveness and sustainability of the interventions. It covers the inception period and the first phase of implementation which took place between November 2018 and December 2020. It is primarily aimed at GEC project partners, wider implementors and non-governmental organisations interested in understanding VfM in education programming within a similar context.1 The findings within this brief build on the evaluation of the Aarambha project and are specific to this project. However, the recommendations are more generic and are aimed at practitioners and others interested in future programming within a similar context.

1. Introduction

The Aarambha project aims to support up to 8,500 married, out-of-school, adolescent girls from the central Terai region, one of the poorest regions of Nepal, with the highest rates of illiteracy, innumeracy, and early marriage. The project is running from 2018 to 2024.

The project reached 1,709 girls in the first year of implementation, addressing low social status, especially for girls from Muslims and Dalit groups. Low social status is the main barrier impeding girls’ safety, health and access to education.

The project activities are centred on the provision of literacy, numeracy and life skills through Community Learning Centres.

In response to COVID-19, the project also started implementing distance teaching and learning activities using mobile phones. In addition, the project has been working with community members (Change Champions) to engage parents, communities and local governments through parental workshops and radio programmes aimed at addressing harmful social norms and raising awareness of the importance of educating girls.

© Sajana Shrestha, People in Need

1 It should be noted that the context for the first year of this project was unique. It had to adapt significantly due to the impact of COVID-19, including school closures etc. The VfM expectations have changed over the course of the first year of operations due to factors outside the project’s control.

THE GEC VFM FRAMEWORK

When reviewing VfM for GEC projects, the GEC VfM framework is applied systematically, drawing on evidence from the endline evaluation findings and project staff interviews. The VfM framework uses four of the OECD DAC criteria:

1. Relevance – whether the project invested in the right activities and modalities to respond to the needs and barriers of the girls identified, with optimal resources allocated to them.

2. (Cost)-effectiveness – whether the project produced the expected outcomes at an optimal cost for the girls and others reached by the project.

3. Efficiency – whether the project delivery was on time, on budget and with good quality processes.

4. Sustainability – whether there has been a long-term continuation of outcomes for the girls and others reached by the project and whether there has been replication and/or scale-up or adoption without FCDO funding.

Read more about the GEC VfM framework and review methodology here

KEY FINDINGS FROM THE VFM REVIEW 2

• To date, the Aarambha project has generated some good VfM and offers strong potential as the project continues to operate. Specifically, the Community Learning Classes and distance teaching and learning activities have offered good VfM – being highly relevant in addressing parental barriers and cost-effective in leading to learning gains in flexible combined modalities despite disruption brought about by COVID-19.

• The radio programmes responding to COVID-19 also appear to be good value for money. However, although they have been highly economical (£3 per girl) and relevant and created positive shifts in attitudes around social norms it is too soon to assess their ability to shift behaviours.

• This project supports some young women who move to inlaws’ houses and may be vulnerable to gender-based violence. To improve their welfare, it is important to change attitudes to education, work, and marriage. In doing this the VfM of the project is enhanced by sustaining non-formal education pathways for continued education, skills development and local employment. These pathways help break through social norms around child marriage, education and gender-based violence and provide longer-term benefits directly to the girls and future generations. To improve the impact and VfM of the project, the review suggests the following adjustments (set out in order of ease and priority, high dowry prices being the most challenging barrier to shift):

1. Adapting the curriculum to meet girls’ abilities

2. Continuing the hybrid mode of Community Learning Centres and distance teaching and learning activities

3. Continuing radio programmes

4. Adapting parental workshops and Change Champions to optimise VfM

5. Improving re-enrolment by providing greater vocational activities for older girls and addressing the barriers of high dowry prices for older, more educated girls and absenteeism (during harvest times and festivals).

This VfM review draws on evidence from project’s endline evaluation of the first cohort and interviews with the project team. As described in the project baseline evaluation report, the project evaluation undertook a quasi-experimental approach tracking 400 married out-of-school adolescent girls in treatment and comparison groups across each evaluation point of data collection – baseline and cohort endline each year – tracking pathways of learning and transition. In addition, a mixed-method approach has been used comprising a quantitative household survey, qualitative focus group discussions and key informant interviews with various stakeholders.

2. What have the project costs been so far?

The expenditure covers the inception phase (from November 2018 to September 2019) and the project’s first year of implementation (cohort 1 from September 2019 to December 2020). It must be noted that any unit cost analysis overstates the costs (and under-reports VfM) because it includes inception phase costs that should be spread over the duration of the project life cycle. Over time, the average cost per girl should decrease and these figures will be collected to demonstrate this trend.

Figure 1 illustrates the percentage expenditure breakdown by the main interventions.3 The indirect overheads have been fully allocated across activities based on project estimates of staff time spent on activities to indicate the reality on the ground of actual implementation costs. The cost per girl for cohort 1 and inception phase averages £219.

To put this figure cost into context, the average expenditure per student for the academic year 2019/2020 for a secondary school in Nepal was £155. 4 While this is indicative, it is not a like-for-like comparison as the Aarambha project does not operate in the formal school setting. The project also benefits from broader outcomes beyond learning but does not reap the benefits of economies of scale inherent in formal school settings.

2 The purpose of the VfM reviews is not to question project aims and objectives. Instead, it is to examine the evidence from a project to judge the degree to which interventions provided value for money.The reviews also recognise that projects operate in different contexts, which means the findings are not generalisable. Furthermore,VfM is one dimension, among many, by which the relative success of programming can be judged.

3 The expenditure that is analysed in this report covers 68% of expenditure on the main activities. Curriculum support and other learning activities are not included in this expenditure or analysis.

4 Source: Peano 2020 EMIS database. Ministry of Education Science and Technology. ASIP & AWPB (CEHRD 2019c) cited in Peano 2020.

Also, in response to the COVID-19 pandemic, the project had to introduce remote teaching and learning methodologies, potentially differentiating it from formal schooling. And finally, the Aarambha projects supports very marginalised girls, which results in higher unit costs than an average learner in Nepal. This indicates a good equity finding for Aarambha.

The distance teaching and learning classes conducted by the project through mobile phones were a relatively new concept and relevant means of adapting to home-schooling, comprising almost 20% of the total budget. According to the qualitative findings from the endline evaluation, girls and parents found that the telephone learning sessions (three to four sessions per week lasting 10 to 15 minutes) were an effective adaptation to COVID-19 school closures. For some girls, distance teaching and learning classes were more relevant than Community Learning Classes, as indicated by their low attendance for the Community Learning Classes. In the evaluation, girls said that in-laws were more supportive of this mode of education as they could see it taking place. However, for some girls, accessing distance teaching and learning was a challenge as they lacked a strong phone network and access to phones, and struggled to take time away from house chores. A mixture of phone support, one-to-one coaching and weekly home visits would be more relevant in the future.

CLCnumeracy,literacyDTLliteracy,numeracyCLClifeskillChangechampion DTLlifeskillParentworkshopRadioprogrammes

3. Which interventions have been most relevant so far?

Based on the evaluation findings, the Community Learning Classes are so far showing strong relevance in providing literacy and numeracy and life skills coaching for the most marginalised girls. They meet the critical learning needs of these girls and also successfully address the barrier of parents’ fear of girls travelling away from the community. The Community Learning Classes comprise 33% of the budget, the highest cost, which was expected as they were the project’s core aim. The main cost driver was the paid educators rather than infrastructure. The educators were also heavily drawn on by other interventions that need local enablers. The high relevance suggests that this has been a good use of project resources within the implementation context so far.

The endline evaluation found that radio programmes were relevant in response to COVID-19. Radio programmes successfully addressed parental attitudes towards girls’ education and only accounted for 2% of the project expenditure. The evaluation found that half of the girls were aware of the programme and its messages on pregnancy, protection from violence and health.

Life skills classes delivered through the Community Learning Classes and distance teaching activities were very relevant, addressing the social norms barriers relating to girls’ empowerment and family planning and contraception. They comprised 10% of the expenditure to date (6% for Community Learning Classes delivery and 4% through the delivery of distance teaching and learning). But their lack of effectiveness (as discussed below) suggests that some adjustment to the design of this intervention could be necessary to make it more relevant.

In the evaluation, the parental workshops, which comprised 6% of the expenditure so far, displayed mixed relevance. They do target the barriers to schooling but not sufficiently enough, as observed by low enrolment rates in the Community Learning Classes. The project has not yet sufficient time to address the deep rooted barriers of absenteeism during the festive and harvesting seasons and early marriages arranged by parents to avoid higher dowry prices. The project will continue with the parental workshops to address these barriers to schooling, with some adaptations in delivery (introducing oneto-one sessions, for example) to better address these barriers and sharpen relevance.

Other parental barriers regarding education versus vocational training remain. According to qualitative findings, parents thought that providing girls with vocational cash grants for income-generating opportunities was more relevant than providing opportunities for formal learning. This raises the question of whether a project focus on formal education is appropriate for older girls within these communities and whether better use of project resources would be to focus on a more flexible blend of skills development opportunities for older girls to transition access employment opportunities successfully. Also, encouraging girls to move into nontraditional vocational pathways may help change traditional gender stereotypes. Younger girls were more likely to transition to formal schooling, which is relevant for the project in the future.

4. Which interventions have been most costeffective so far?

Literacy and numeracy through Community Learning Classes

The evaluation found significant positive impacts on literacy and numeracy, driven by the fact that the classes were run in the community and parents did not have to worry about the daughters travelling long (potentially dangerous) distances. In addition, the evaluation found that the treatment group demonstrated higher learning gains than the control group and that an extra 25% of girls met the minimum reading proficiency level.

Figure 1: Intervention costs as a percentage of the budget for inception phase and first year of implementation (November 2018 to December 2020)

Source: Aarambha project financial data

However, some girls were still unable to identify words, read, and comprehend the passages of text. In terms of numeracy, some girls could not add, subtract, multiplicate and divide. The evaluation suggested that the project should adapt this intervention by further improving the curriculum and holding separate teaching sessions for groups with different abilities. The project could also consider running longer classes or providing group-based support to improve reading skills and show the practical applications of the learning to market trading and other life skills. This would sharpen relevance and effectiveness. The project could also benefit from continuing the hybrid delivery of learning through Community Learning Classes and distance teaching and learning, which has worked well so far.

The cost per girl for the literacy and numeracy classes for the first year was £106, comparable to the Nepal formal school annual figure (£155), suggesting good cost-effectiveness. However, the evaluation found that re-enrolment to formal education was not effective, especially for older girls. As explained earlier, qualitative evidence from the evaluation suggested that this was due to girls and parents being less interested in formal education and more interested in vocational training due to its focus on income generation opportunities.

Literacy and numeracy through distance teaching and learning

The evaluation reported that almost all girls had attended literacy and numeracy classes through distance teaching and learning activities. Girls said they were satisfied with the distance teaching and learning support provided by the project during the lockdown and classes improved their learning. Girls also reported that they got the necessary support from parents during home-based learning. The cost per girl was £20 for literacy and numeracy. This suggests good cost-effectiveness in comparison with the unit costs of the Community Learning Classes. But it must be noted that the distance teaching and learning component benefited from the sunk costs of investment coming from the creation of and enrolment in Community Learning Classes. In the future, even without lockdowns, the project could benefit from continuing the hybrid model that combines Community Learning Classes and distance teaching and learning activities to maximise cost-effectiveness and relevance.

Life skills classes through Community Learning Classes and distance teaching and learning

The cost per girl for life skills classes was £20 for Community Learning Classes and £18 for distance teaching and learning. According to the evaluation, the adolescent sexual and reproductive health classes delivered through the Community Learning Classes led to shifts in attitudes and behaviour for matters related to family planning and contraception. However, the coaching activities aimed at increasing girls’ self-esteem did not change behaviour, suggesting that girls still had no voice within the household on school enrolment or starting their own business and that elders have the final say on those matters. It takes time to address deep-seated social norms, so after just one year of project activities, it is not surprising that these barriers still exist.

Longitudinal research that tracks young beneficiaries is needed to understand the longer-term impact of project activities on attitudes and practice towards education, early marriage and empowerment.

Change Champions

The project appointed 50 community leaders as Change Champions to help break down deep-seated social norms. Change Champions conducted household visits aimed to raise awareness of the importance of educating girls amongst parents. They also engaged with local government officials advocating for girls’ education. The resistance to change in community attitudes and behaviours suggests that this intervention has not yet had a significant impact. This activity comprised 5% of the budget, at the cost of £526 per Change Champion. This relatively high cost reflects the high fixed project inception costs, which will be spread over the following years. Thus the unit cost is expected to decline in future years.

In the future, the project could consider blending the life skills components and the Change Champions. More meetings with the Change champions could also improve effectiveness by reinforcing the messages and reaching a larger number of parents to effect greater change.

Parental workshops

Whilst the quantitative evidence from the evaluation found emerging changes in parental attitudes towards girls’ education, qualitative evidence suggested that shifts in attitudes have not yet changed actions or behaviours. Parental workshops have not yet shown strong effectiveness. As above, it may be too early to expect behaviour changes in the first year of implementation. The project could benefit from adapting the content or delivery of this intervention, for example, experimenting with alternative behaviour change models.

Radio programmes

Radio programmes were effective in changing parents’ attitudes towards girls’ education during the COVID-19 pandemic. There is some tentative evidence from the evaluation to suggest that they were driving the decrease in time spent on household chores. Beyond that, it is unclear whether they have shifted attitudes. Again, it may be too early to tell. Their cost was low at only £3 per beneficiary (based on 3,259 girls and their families), which was very economical. Nevertheless, it is recommended to continue such programmes not only to respond to the current pandemic but as another way to address barriers presented by social norms. Moreover, assessing change in knowledge, attitudes and practices might be more relevant in judging the results generated by the combined (complementary and mutually reinforcing) radio programmes and parental workshops.

5. Which interventions have been showing sustainability so far?

There has been an encouraging commitment by the local government towards sustaining the project activities. Some municipalities have provided written commitments to continue the Community Learning Classes in the future. Since the project is working across various municipalities of the same district, it should continue with its systemlevel activities within the next phase. The project is also involved in the Sisters’ for Sisters education network, a girl-led initiative supporting girls’ education across the GEC projects in Nepal.

Partnerships with the government could be strengthened for more cost-effective components, such as radio programmes. The project could also work with the government for hybrid

approaches of distance teaching and learning and Community Learning Classes. This may lead to employment opportunities for the local facilitators trained by the project by the Sisters’ for Sisters Education network. Political will and available budget do, however, remain a challenge and a potential barrier.

At the household level, challenges remain for sustaining learning as no parents are happy to send their daughters away for higher studies. The project could address this challenge by ensuring girls can learn within their community using technology or distance learning approaches to continue and sustain learning. Alternatively, the project could focus more extensively on working with parents to assuage their concerns, encouraging younger siblings to start school, offering vocational activities for Aarambha graduates and helping to reduce the prevalence of child marriage.

6. Conclusions

Findings from the VfM review suggest that the project has shown good VfM so far, providing a strong foundation. To illustrate the VfM of the specific interventions achieved so far, Figure 2 plots the intervention costs against the benefits delivered, as proxied by effectiveness, sustainability and relevance. The top left quadrant, where a high value of benefits is delivered through a low-cost investment, shows the highest VfM. Radio programmes have the highest VfM so far, followed by the blended approach of distance teaching and learning and Community Learning Classes. However, the project overall shows good potential and there is scope for interventions to veer to the top left as the project progresses.

Figure 2: Investment costs versus benefits of each intervention

The benefits are further broken down and rated against each of the VfM review criteria.

Table 1: VfM ratings for interventions against VfM review criteria

5 This review covers the period from inception to the end of the first year of implementation. During this period 68% of the project budget has been used, so only this expenditure is considered in this review. The remaining 32% of the budget has not been included in this report.

Annex 1: GEC VfM review methodology

The aim of the GEC VfM framework is to offer a quick pragmatic methodology to review the VfM of a GEC project by using existing evaluation findings. The framework uses the OECD DAC criteria (relevance, efficiency, effectiveness and sustainability) and evaluation findings, and reframes them through a VfM “lens” drawing out the key features of the findings that point to strong efficient value generation for the right people against optimal costs and resource allocation.

Effectiveness

The GEC approach to VfM analysis relies on the extraction of effectiveness data from the evaluation reports (midline and endline). This should include all the different types of outcomes assessed (learning, transition and sustainability) and the intermediate outcomes, such as wellbeing and life skills, self-esteem, and social norms and behaviour changes. It should also include data on how effective interventions have been for different targeted subgroups. All types of data used to demonstrate effectiveness are relevant for VfM purposes (quantitative or qualitative data, including the beneficiaries’ voices on what they found most valuable).

Effectiveness can either be assessed for a GEC project as a whole, or for separate components. Some projects’ midlines or endlines may be able to disentangle the impact and causality of certain interventions on outcomes over and above others. This likely will only be feasible for evaluations with a comparison group.

Cost-effectiveness

With activity-based budgeting, specific interventions can be assessed on cost-effectiveness. Costs can be presented in cost per girl format, with narrative attached to it, explaining what the overall cost per girl achieved in terms of outcomes observed. The number of girls reached by interventions can differ, thus giving rise to very wide-ranging figures. These variances should be discussed within the VfM analysis. Benchmarking to similar projects within the same context would be useful.

If there are strong, statistically significant findings, with a control group of girls displaying the counterfactual ‘without project’ learning achieved in a year of schooling, the analysis can be taken further to estimate the Cost-Effectiveness Ratio (CER): additional years of schooling per $100 spent.

Relevance

The GEC approach to VfM and relevance is to use evaluation report findings around relevance to understand whether resources were allocated to meet the needs and objectives identified upfront, and whether optimal resources were allocated across activities. If such needs changed over time, the VfM review should consider whether resources were reallocated to reflect this to maintain relevance of the programme.

An equity angle to relevance would determine whether or not the right beneficiaries were targeted by the project according to needs, and if enough or optimal resources were allocated to various targeted groups. The analysis should consider whether, in retrospect, the budget would have been carved differently across activities to reflect relevance better, and whether there was a correct allocation of funds for technical functions, monitoring and evaluation, and management etc.

Sustainability

Sustainability within the GEC is measured by:

1. A long-term continuation of outcomes for the direct beneficiaries themselves (including targeted schools and communities) and

2. Replication and/or scale-up or adoption of project activities without the need for FCDO funding.

A project may have sound input costs (Economy), have a demonstrated ability to translate its activities into quality outputs (Efficiency), and achieve its targets with regard to learning and attendance (Effectiveness), but may not have a strong sustainability case. Sustainability is not always covered in the conventional measures of VfM. But it is another factor to justify expenditure. The evidence of evaluation findings on contributions to sustainability should be integrated into a VfM narrative. Evidence of replication or scale up beyond project funding would point to very strong VfM. Sustainability intent may have been present from the start in the form of specific design features or plans. But over time, as contexts have changed, contributions to sustainability may not have materialised. This may require projects to undertake additional activities targeting sustainability that increase their costs but do not necessarily improve their efficiency or effectiveness in the short term. These should be considered in a VfM assessment. Another angle to considering costs and sustainability is defining the minimum spend for activities required to achieve sustainable outcomes. For example, determining the cost of a minimum amount of project exposure/duration or intensity necessary to achieve sustained outcomes based on findings.

Efficiency

Taking a narrative approach, assessing efficiency involves understanding how smoothly processes and interventions have been delivered (speed, quality, cost). There are four aspects to efficiency:

1. Whether the project as a whole was delivered on time and on budget

2. Assessment of the speed, quality and cost of the operating models for each intervention and the project as a whole

3. Assessing the efficiency of processes and management of the project as a whole

4. Assessing the efficiency of targeting girls (inclusion or exclusion errors).

Limitations of the GEC VfM approach

The VfM analysis is dependent on the evaluation findings, so its efficacy depends on the efficacy of the evaluation findings. It is also dependent on the ability of projects to producing expenditure data in relevant formats (activity-based budgeting). VfM assessments such as these are not often appropriate for making comparisons with other projects, due to differing contexts, cost structures and activities.

For more information, contact: learningteam@girlseducationchallenge.org | www.girlseducationchallenge.org

The Girls’ Education Challenge is a project funded by the UK’s Foreign, Commonwealth and Development Office (“FCDO”), formerly the Department for International Development (“DFID”), and is led and administered by PricewaterhouseCoopers LLP and Mott MacDonald (trading as Cambridge Education), working with organisations including Nathan Associates London Ltd. and Social Development Direct Ltd. This publication has been prepared for general guidance on matters of interest only and does not constitute professional advice. You should not act upon the information contained in this publication without obtaining specific professional advice. No representation or warranty (express or implied) is given as to the accuracy or completeness of the information contained in this publication, and, to the extent permitted by law, PricewaterhouseCoopers LLP and the other entities managing the Girls’ Education Challenge (as listed above) do not accept or assume any liability, responsibility or duty of care for any consequences of you or anyone else acting, or refraining to act, in reliance on the information contained in this publication or for any decision based on it.