13 minute read
4. Monitoring and Evaluation for Results and Impact
This section has explained how additional scrutiny on risk and risk management ensured VSCF project implementation throughout a period fraught with multiple risks. Lessons to be drawn for future programming include:
• Enhance risk management processes as needed, and as contexts change. • Clarify risk appetite with partners from the outset, ensure all partners are informed about risks, understand each other’s perceptions of risk, and collaborate on risk management. • Agree timely, accurate and useful risk reporting to enhance decision making and support management and oversight. • Use risk scenario planning to anticipate challenges and choose paths that can be managed.
This section describes how the Facility Manager’s approach to monitoring and evaluation contributed to the VSCF’s achievements. It highlights how a focus on monitoring for impact and fast-tracking of data collection, enabled responsive decision-making by programme partners, informed programme direction and maximised value.
The effectiveness of monitoring for ‘Results and Impact’ is evidenced by:
• BP4GG was awarded an A+ in the Annual Review 2020 and scored 76 on the FCDO Strategic Relationship
Management scorecard - exceeding expectations. • Increased reach from an initial target of reaching 200,000 people to reaching 1.4 million people by the end of the programme. • Increased measures of VfM by a factor of 20 (4000 workers at £42.50 per beneficiary to 72,496 workers at £2.10 per beneficiary) by redirecting resources following the coup in Myanmar to the Bangladesh garments projects. • Over achievement of outcome targets: 11 times more beneficiaries reporting changed workplace practices than the original target set. • Over achievement of impact targets: 9 times more beneficiaries having improved livelihoods than the original target set.
Build the Theory of Change and logframe together, creating opportunities for ownership of the full theory of change and assumptions that underpin the change process. When FCDO decided to pivot the BP4GG programme and establish a COVID-19 rapid response Facility, the programme’s theory of change, and assumptions underpinning it, were revised. This created a unique opportunity for implementing partners to influence the full theory of change, which in turn created buy-in and ownership of the selected indicators of change used in the logframe.
The theory of change reflected a ‘shared value perspective’ i.e., if partnership projects delivered social and economic interventions for vulnerable groups to help them deal with the impact of the pandemic, these initiatives would also support businesses to continue their operations in a safer and more responsible way.
As it was impossible to predict what could realistically be achieved by the VSCF given the uncertainty of the pandemic, VSCF logframe targets were set with caution, whilst attempting to be realistic when ‘predicting the unpredictable’.
Focus upon big picture outcomes - not inputs. Given the volatility of the COVID-19 situation and the likelihood activities would be disrupted, the Facility Manager focused on the overall value, and results to be achieved at outcome and impact levels, rather than fixating on activity and output level targets. This reflected the importance of targeted impact to ensure the VSCF delivered value for money.
Managing Value for Money. Throughout implementation of the 8 chosen projects, the Facility Manager encouraged projects to improve value for money (VfM). The VfM improvements were managed in several ways, including through the identification of additional beneficiaries, and by adapting activities for greater impact. As a result, targets for indicators of VfM set at the beginning of the VSCF projects were exceeded. By the end of the programme, the cost per beneficiary reached with programme products and services (Output level) was £1.69 compared to a target of £11.87. At Outcome level, the cost per beneficiary reporting improved practices was £2.03 compared to the logframe target indicator of £23.75. At Impact level, the cost per beneficiary was £3.23 compared to the logframe target indicator of £29.68.
Prioritising women and marginalised groups: ‘Equity’ was explicitly prioritised throughout each of the projects. Women and marginalised groups were targeted beneficiaries during design and implementation which resulted in excellent VfM results. At the end of the programme 55% beneficiaries were women and girls. It costs FCDO £2.97 (a fraction of the target of £24.48) to identify a women worker, £3.05 to reach her (against a target of £24.48), and £3.67 to help her improve practices.
To ensure that resources were equitably distributed, and reached those most in need, the VSCF surveyed the socio-economic status of beneficiaries. VSCF projects beneficiaries included 36% youth and 3% People Living with Disabilities. The endline surveys carried out in June 2021 showed that 63% of beneficiaries lived below $5.50 (PPI 2011), 38% below $3.20 a day, and 13% below $1.90, confirming that BP4GG had successfully reached those most in need. In terms of gender, the VSCF target of 50% female was also exceeded. At the end of the programme 55% beneficiaries were women and girls.
Creating feedback loops through collecting beneficiary feedback to inform decision making. The VSCF established several methods to collect feedback from beneficiaries and generate data on how effectively projects were responding to need and maximising impact.
In the context of the rapid delivery of the VSCF and the lack of opportunities to travel and visit projects on the ground, the Facility Manager needed to find alternative means of real time, reliable data collection. The lean data supplier 60 Decibels was contracted to collect data quickly and safely, directly from beneficiaries. The supplier conducted 14 surveys. Seven were conducted mid-way through projects to capture early signs of impact on beneficiaries, two were completed at the end of the grants period and five were post-grant studies to capture impact beyond the VSCF funding period.
“The benefits seen of the union-led IGAs (Income Generating Activities) included real diversification, real experimentation. I was surprised at how many types of IGA people wanted to try. It was important that we bought that risk, so they could see which IGAs would work. After consultation the unions came with what they really wanted to try – the main take away was that the grants supported the unions to experiment.”
Andrew Parker, Imani Development, Cadbury Wrap Up Workshop
A combination of remote data collection tools was used. Data were collected via mobile phones using SMS and Integrated Voice Recordings [IVRs]. To encourage participation, supplier factories informed their workers of feedback requests via their mobiles and reimbursed the costs incurred from survey completion. Worker data was protected through 60 Decibels data protection system removing all personal information so that feedback remained anonymous.
Feedback data also enabled partners to adapt project activities. For example, the ETI ‘Securing workers’ rights in a COVID-19 context in East African agriculture supply chains’ project asked workers which training methods and communication messages were most useful for absorbing COVID-19 health and safety measures. In Kenya, beneficiary feedback found training to be effective whilst in Zimbabwe responders said posters were the most useful learning tools. ETI used this feedback to prioritise the means of communication to the local context to maximise impact and reach many more beneficiaries with important health messaging.
Beneficiary voices provided vital sources of data to evaluate whether VSCF projects were successful in reducing the vulnerability of people to COVID-19, in supporting businesses to recover from the pandemic and livelihoods to become more resilient. Beneficiary Feedback surveys conducted in June 2021 confirmed that VSCF projects were achieving the desired results, as:
1. 93% of beneficiaries surveyed reported the support they received as useful. 2. 85% reported their ability to cope with the pandemic has been improved. 3. 96% reported workplace changes where employers have implemented changes to workplaces to make these safer. 4. 92% reported satisfaction with workplace improvements. 5. 92% reported that their sense of safety and security at work has been much improved. 6. 63% reported improved confidence in earning incomes.
Prioritising Social Inclusion and Climate Resilience. The scoping research completed to inform the design of the VSCF identified the garment and agriculture sector supply chains contained a concentration of women and vulnerable workers³ who were disproportionately shouldering the burden of the impact of COVID-19. Ensuring equity of benefits was at the heart of the VSCF which meant it also had to be at the heart of its MEL processes and systems. For example, the Fairtrade and Mondelez International Cadbury Farmer Resilience Fund project was designed to have a 25% uplift for each female member. At the end of the project 82% of those earning incomes from non-crop grant activities were women. The ‘Bridging the Gap: Supporting the transition from crisis to resilience’ project in Myanmar supported over 1,500 workers (98% women, an increase from the original target of 92% women) in with cash transfers in the form of ‘two weeks’ stipends. People with Disabilities and young people also reported benefits from project activities.
Protecting the vulnerable from climate shocks and supporting people most impacted by the pandemic to become more resilient was central to the VSCF. The Cadbury Farmer Resilience Fund project awarded income generating grants for activities specifically suitable to the local climate. Over 7,000 farmers received training on agronomy, Good Agricultural Practices and conservation agriculture embedded in community led planning in support of climate mitigation and adaptation.
Summary of lessons for future programming
“One of our strategic plans or objectives is to expand or strengthen the livelihood activities that our women and youth are doing at the various societies. However, [having] the funds to do that at this COVID-19 pandemic moment was a challenge. So the timely intervention by this project has helped the Union to [support] our women and youths who are jobless and this has completely added a face lift in our societies.” Fanteakwa Union, Ghana
This section has described how the VSCF monitoring and evaluation system was designed to enable responsive decision making and to maximise results. The findings suggest that future programmes would benefit from:
• Using the co-creation phase to set the MEL plans with partners helped smooth implementation. • Allowing the lead supplier to update the programme theory of change and logframe at the outset of the programme, to strengthen ownership and understanding of the theory of change, and commitment to the logframe indicators. • Focusing on outcomes and impact from the outset and demonstrating flexibility in inputs and activities. • Creating feedback loops so that data on results achieved are collected early enough, and directly linked to decision making processes. • Using beneficiary feedback surveys to provide data for evaluating the effectiveness and impact of interventions.
This section describes how the VSCF captured evidence, and disseminated learnings generated by VSCF projects for longer-term sustainability. Learning mechanisms were established for partners to improve impact at project level. To encourage replication and scale beyond the project teams, lessons learned were widely communicated with external partners. Given the innovative nature of VSCF partnerships and projects, a key function of the Facility Manager was to harvest insights offered by VSCF projects into new ways of working and about the effectiveness of pilot interventions, to create evidence, and to share this even more widely, including to UK businesses, the wider UK government, and broader professional community.
The effectiveness of ‘Communication and Learning with purpose’ is demonstrated by:
• A shared vision of the communication strategy built with partners and supported by effective tools (Strategic
Communications Guidelines, quarterly Strategic communication meetings, Strategic messaging matrix and a communications calendar). • Communication and learning activities were delivered for a specific purpose, with messaging, ‘hooks’ for events and products. • Communication campaigns on specific themes were designed around global milestones, with events held and relevant products released to coincide with the global milestone, VSCF learning events for e.g., women’s day, G7 meeting (build back better) and COP 26 (climate resilience). • Leveraging lessons from individual projects to aggregate findings and disseminate widely, through the 35 products produced by the Facility Manager and consortium partners (8 one-page project summaries; 10 Learning Briefs; 4 Case
Studies; 1 Business Case; 3 High Level Event summary read outs; 4 Quarterly Newsletters; 4 videos on topical themes; an end of project brochure reaching over 178,000 people). • 2 Business roundtables for safe and open discussion between business and FCDO with 60+ participants. • 20 learning events were organised by the Facility Manager including technical sessions across portfolios and high-level events marking international milestones attracting over 1600 participants globally. • Partners also published 24 knowledge products and organised 32 learning events reaching 1769 participants. • High impact comms in collaboration with FCDO country offices. The FM worked closely with the BHC Kenya to provide content for the article in the Economist on 13 Feb 2021 and the Economist Intelligence Unit podcast (Exit-stage plight:
Brexit’s costs come due - Economist Radio | Podcast on Spotify) which focused on the flower projects in Kenya timed with BHC visit to the project sites. • Dissemination events held to communicate effective VSCF innovations to broader sector stakeholders. For example, support provided to the Kenyan Flower Council to hold an event on greening exports from Kenya through sea freight, attended by the BHC and Embassy of the Netherlands, at which findings of VSCF project shared and discussed.
Recognising learning and communication are important pathways in the theory of change. Recognising the vital role that learning and effective communications play within processes of social change, learning and dissemination were integral to the VSCF theory of change. These components of work were included in the logframe as a separate output, and VSCF projects included resources for learning and communication activities.
“Mott McDonald has been very directive in strategic communications management. We had been anticipating negative press. Their response was proactive. This was not a big programme, but it made considerable achievements in 12 months which were communicated very effectively across a range of media channels including targeting the private sector to encourage uptake of learning. There is real merit in the learning sessions. Many development programmes fall into a trap of becoming insular, only focussing on logframe targets and not understanding the value of external communication and lesson learning.” Raania Rizvi, Lead Adviser (August 2021-December 2021), FCDO
As a result, the VSCF spurred the development of a wide variety of learning and advocacy resources. For example, Fairtrade and MM Flowers developed the UK Covenant for the flowers sector which is being used for advocacy internationally and in the UK. ETI and Partner Africa prepared resources for a campaign on working conditions in East Africa. In Bangladesh, ETI created a free online learning resource (project materials and user guides) to help additional brands and factories implement Occupational Health and Safety (OHS) initiatives in the future. In Kenya, the Facility Manager supported the Kenyan Flower Council to hold a high-level learning event with participation from the British High Commission and Kingdom of the Netherlands. The event showcased evidence produced by the Sea Freight Flowers project led by Flamingo Horticulture, that demonstrated that sea freight is sustainable, scalable, and cost-effective route to export produce from Kenya to UK, contributing to economic growth and reducing carbon emissions. The event was attended by over 140 participants from the flower sector, including senior members of the Kenyan government and private sector players. Strategic messaging across multiple communication formats. The Facility Manager prepared a messaging matrix to plan dissemination of learning about themes, across a library of publications. The messaging matrix is presented in Table 2, below.
“The pilot sea freight flowers project has highlighted the value and viability of sea freight transportation not only of cut flowers but other Kenyan produce. Sea freight offers great potential for economic growth and job creation as well as supports Kenya’s commitment to 0% emissions made at COP26. We are committed to supporting the infrastructure at the port and the SGR to prioritise sea freight expansion.” Betty Maina, Cabinet Secretary Ministry of Industrialization, Trade and
