Supporting community action on AIDS in developing countries
Measuring and Improving the Value for Money of HIV Programming: The Approach of the International HIV/AIDS Alliance
A Discussion Paper December 2010
DRAFT Š International HIV/AIDS Alliance, 2010. Information contained in this publication may be freely reproduced, published or otherwise used without permission from the Alliance. However, the Alliance request that they be cited as the source of the information. For more information about Alliance publications, please go to www.aidsalliance.org Published December 2010 Registered charity number 1038860
Measuring and Improving the Value for Money of HIV Programming 1
DRAFT Background and rationale Over the past year the Alliance has been developing a growing interest and commitment to the issue of value for money, by firstly defining its approach at an organisation wide level and secondly implementing robust value for money practices in its programming. The recent evaluation of the International HIV/AIDS Alliance’s (the Alliance) Programme Partnership Arrangement (PPA) with the Department for International Development (DFID) included a section analysing the value for money of the Alliance’s worki. The evaluation recommended that ‘it would be helpful for the Alliance to produce a document analysing value for money issues related to its approach and programmes.’ This document has been produced in response to that recommendation.
Why is value for money important?
Box 1: Areas of focus for the new UK Coalition Government ‘In July, we published our ... objectives on international development, making it clear that the emphasis will be on results – about what is being achieved on the ground. I am acutely conscious that in these difficult economic times we must demonstrate to taxpayers that we are getting value for money and I have launched full-scale reviews of where that money is going and how it is being spent. These reviews will give us evidence from our experience in the field combined with the views of a wide range of people and organisations, which will allow us to make informed decisions about our future funding.’ Statement made by Andrew Mitchell, Secretary of State for International Development. Making the Case for UK Aid available on http://www.dfid.gov.uk/ Media-Room/Speeches-and-articles/2010/ Message-from-Andrew-Mitchell/
The context of financing responses to HIV in low- and middle-income countries has changed dramatically in recent years. Two significant contextual factors are introduced here. First, the global financial crisis means that the dramatic growth in financing for HIV responses seen following the UNGASS declaration of 2001 is not being sustained. There is a need for those managing responses to HIV to focus on ensuring value for money and ‘enhance the impact of current investments by improving the efficiency, effectiveness and quality of programmes, strengthening linkages between programmes and building systems for a sustainable responseii. This has been referred to by some as making the money ‘work harder’iii. Second, there have been changes in public health and development priorities at national and international levelsii. Concerns that weaknesses in health systems, including shortages of health workers and weak health management information systems, are undermining efforts to scale up HIV-related services and have resulted in an increasing focus on strengthening health systems to provide HIV-related services rather than only focusing directly on delivery of those services. Consequently, many international agencies are focusing strongly on ensuring value for money. For example, the Global Fund to Fight AIDS, Tuberculosis and Malaria has emphasised the importance it places on value for moneyiv,v, UNAIDS has produced technical guidance on how to ensure value for money in Global Fund proposals. The new coalition government in the UK, in particular, has stressed that given the current difficult economic times they will intensify the UK’s focus on ensuring it achieves value for money (see Box 1)vii. There are a number of reasons why the Alliance is concerned about value for money. The Alliance wants to be more effective and to achieve more with the funds available to it. It wants to use value for money data for decision-making and to demonstrate that its approach is credible. The Alliance also wants to be responsive to the needs of its funders and to be able to engage in meaningful discourse with them. But, ultimately it is about getting more value for the beneficiaries of the Alliance’s programmes. Much of the current discourse about value for money is premised on concerns relating to the providers of international aid, both as institutions and as individual taxpayers and contributors. For example, this perspective is clearly seen in the current interest in initiatives such as results-based aid, results-based financing and
Measuring and Improving the Value for Money of HIV Programming 2
DRAFT The Alliance believes that the end-user perspective on value for money is of critical importance and needs to be highlighted in discussions on the topic. Beneficiaries and recipients of aid have a key interest in value for money. They are often concerned to ensure that funds reach them, that waste is minimised and that funds are used to provide the services that they need and prioritise.
results-based management. These initiatives seek to link payments more directly to the achievement of particular results with the intention of ensuring greater value for money of the funds provided. However, it is not only donors that are concerned about value for money of aid.
What do we mean by value for money? The term ‘value for money’ is used in a variety of different ways in different situations and contexts. Broadly it seeks to judge the value of particular outputs, outcomes or results against the cost of achieving them. This means that value for money can be increased in two main ways – either increasing value or reducing costs. Of course, a combination of both of these would result in a particularly sharp increase in value for money. This has been termed ‘doing more with less’. The Alliance has outlined a conceptual framework for measuring value for money which is structured around a ‘results chain’ (see Figure 1). This results chain sees inputs resulting in activities which produce outputs which contribute to outcomes and impact. Measurements can be made at a number of stages of this chain. • Costings are essentially a measure of financial input. • Cost efficiency measurements relate to the translation of inputs into outputs through activities. • Measures of effectiveness focus on whether outputs produce desired outcomes and impact. • Measures of cost effectiveness relate costs to the outcomes achieved. Demonstrating value for money may involve measurements at all of these levels. Figure 1: Conceptual framework for the VfM project
Cost efficiency Inputs
Activities
Effectiveness Outputs
Outcomes
Impact
Cost effectiveness
Costings and Cost Efficiency Given the contextual drivers of the current discourse on value for money in HIV programming, in general, and the reducing availability of funds, in particular, it is unsurprising that much of the focus of discussions around value for money is on reducing costs. A tool for doing this is to identify and reduce the ‘unit costs’ of delivering a service. Comparisons may be made with historic performance, ie what it cost previously to deliver that service, and what it costs others to provide the same or similar services. Work on analysing unit costs of HIV services has shown that these tend to reduce as services expand in scale but that there is very wide variation between and within countries. For example, a non Alliance study determined costs of voluntary counselling and testing to be $668 in Mexico and only $1.50 in Russia. Variations within the five countries studied varied between 10 and 100-fold. The Alliance has similar data which shows very wide variation in the unit
Measuring and Improving the Value for Money of HIV Programming 3
DRAFT Box 2: The National Institute for Clinical Excellence (NICE) and Value for Money The National Institute for Clinical Excellent (NICE) is an independent organisation responsible for providing national guidance on promoting good health and preventing and treating ill health. NICE provides guidance, sets quality standards and manages a national database to improve people’s health and prevent and treat ill health. NICE makes recommendations to the NHS on new and existing medicines, treatments and procedures; and treating and caring for people with specific diseases and conditions. NICE makes recommendations to the NHS, local authorities and other organisations in the public, private, voluntary and community sectors on how to improve people’s health and prevent illness and disease. Using NICE guidance may reduce costs while at the same time maintaining and even improving services. NICE oversees the development of indicators which are used as part of the Quality and Outcomes Framework which is a performance-based, incentive scheme for GPs in the UK. NICE has a guidelines manual which includes a section on incorporating health economics in guidelines and assessing resource impact. NICE draws heavily on the use of qualityadjusted life years (QALYs) in its economic assessments. Although there is no particular cut-off between cost-effectiveness and costineffectiveness, the consensus among NICE’s economic advisers is that NICE should accept as cost-effective those interventions with a cost-effectiveness ratio of less than £20,000 per QALY. However, NICE’s reliance on QALYs has been criticised as discriminating against the elderly. Similar bodies in other countries either use QALYs less (eg New Zealand) or not at all (eg Canada). NICE has an international division that provides advice to other countries on how to ensure their health practices are as effective and cost effective as possible. Since 2008, NICE International has, with support from DFID provided advice to governments in Jordan, Colombia and Ghana.
costs of prevention services provided, eg between Latin America and the Caribbean and Uganda. Although one explanation of these variations is that lower unit costs demonstrate more efficient provision of services, it is also possible that there is not like for like comparison in terms of value. For example, the low unit costs of VCT in Russia seem to reflect, in part, the very limited amount of counselling provided. There are concerns that attempts to compare unit costs will be invalidated if the services compared are different and produce differing results.
Cost Effectiveness One of the key questions is whether the variations in services provided produce difference in terms of tangible benefits or results. For example, is a longer period of counselling with an HIV test ‘better’ than a shorter period? Does it result in more changes in behaviour, more sustained behaviour change or greater uptake of treatment? These questions really relate to the quality and effectiveness of particular interventions and their variations. An area that requires attention is how adherence to defined quality standardsx, which is necessary for any cost effectiveness analysis, would be assessed. There are particular concerns that some results and benefits of some services could be overlooked by unit cost analyses, particularly where those results are relatively intangible, eg influence on policy. One area of interest in this regard is the result of investment in HIV programming on strengthening health systems and whether this should be considered another benefit to be considered in any value for money analysis.
Broadening the view of ‘value for money’ beyond specific interventions Undoubtedly, much of the discussion and analysis of value for money will be focused on particular interventions, eg what makes these interventions most effective and how can the unit costs be reduced? But, value for money analysis can be applied more broadly beyond analysis of single interventions. The Alliance is interested in conducting robust analyses of the value for money of: • Its overall approach and the Alliance model of building capacity of national-level, civil society, linking organisations (LOs) to respond to HIV • Its current strategy (2010 - 2012) ‘HIV and Healthy Communities’xi • Different financing mechanisms – there is particular interest in considering the value for money of more strategic, unrestricted forms of donor funding, eg as provided through the DFID PPA There are many examples of the use of value for money analyses for decisionmaking. One of the best-known is the National Institute for Clinical Excellence (NICE) in the UK (see Box 2)xii, xiii, xiv.
Measuring and Improving the Value for Money of HIV Programming 4
DRAFT What has the International HIV/AIDS Alliance done to track value for money? The Alliance has undertaken a number of different studies to assess the value for money of its work. These can be broadly mapped on to the conceptual framework identified in Figure 1.
Measuring Cost Efficiency The Alliance Secretariat has conducted some preliminary analysis of the financial expenditure of linking organisations in different countries comparing this to the number of people reached with services. Calculations are based on direct and indirect beneficiaries of programmes. A weighting factor is applied to those reached indirectly. This analysis generated the following data: •
Total expenditure per person reached – in raw terms this varied significantly depending on the type of interventions within a programme eg information education and communication prevention (IEC) interventions vs care and support. A range of extremes from $0.30 to $175.34 were calculated.
• Cost per beneficiary for particular services, such as prevention or care and support. Figures relating to per capita government expenditure on health were included for comparison purposes. Considerable challenges were faced in conducting this exercise including: • Gaps in availability of financial data. • Limited experience of conducting analysis of this nature in NGOs. • Significant variations in the types of projects supported by different linking organisations.
In the Burkina Faso analysis services considered included educational support for orphans and vulnerable children (OVC), health care services for OVC, social services for OVC, community support for OVC, communication for social and behaviour change, condom provision, prevention of mother-to-child transmission, voluntary counselling and testing, training of staff, programme management and administration, social protection and social services, enabling environment, nutritional support for people receiving ART and psychological treatment and support services for PLHIV.
Figures generated from this exercise have been used as the basis for a value for money discussion within the Alliance’s field programmes. This focused on seeking to understand the differences that exist between regions and countries. The Futures Institute has conducted an analysis of the unit costs of Initiative Privée et Communautaire (IPC) Contre le Sida au Burkina Fasoxv. IPC is an Alliance linking organisation. This analysis looked at the unit costs of providing different services through different funding sources. Very wide variations in unit cost were observed depending on funding source. However, like was not always being compared with like. For example on health care for OVC, some funders supported individual children while others supported clubs benefitting multiple children. In those cases, the unit of analysis was the club which explains why the unit cost was higher than for other funders. Further work is needed to define unit costs more precisely and how they should be measured and to understand and explain variations in unit cost for similar services.
Measuring Cost Effectiveness In addition, the Alliance conducted a value for money case study (Box 3) as part of the recent PPA evaluation, using a method referred to as Social Return on Investment (SROI)xvi. The Alliance was interested in testing SROI as a method because of its
Measuring and Improving the Value for Money of HIV Programming 5
DRAFT Box 3: Preliminary findings from SROI case study in India This case study focused on the work of CHAHA (Children Affected by HIV/AIDS) in India. CHAHA provides a wide range of services to children affected by HIV and their families. Some services are provided directly, eg nutrition support, household support, education support, income-generating projects and vocational training. Some services are provided through referral, eg access to antiretroviral therapy and treatment for opportunistic infections. More than 70 beneficiaries were consulted and a range of outputs and outcomes were identified from their perspective. Outputs included receipt of food rations, receipt of school fees, receipt of vocational training etc. Outcomes included improved physical health of children and parents, increased lifespan and improved wage earning opportunities. The incidence of each outcome was calculated or estimated and a value assigned to the outcome. For example, the financial value assigned to improved health status was based on costs of travel for medical services avoided. These financial values were adjusted for ‘deadweight’ – the change that might have occurred anyway without the intervention – and an estimated drop-off in benefit over time. Based on this work, it appears that every £1 invested in CHAHA from 2008 to 2010 generated £6.76 of social, health and financial value. More than half (52%) of this value accrued to parents and caregivers with just under half (45%) benefitting children directly. Most value was created through improvement in livelihood status, improved health status of the child, other children having better income prospects and improved educational attendance and status. Less value was created through avoidance of family crisis, improved health status of the family and improved family care. It is anticipated that this information may be useful to CHAHA and the Alliance in making management decisions, eg on where to focus support.
potential as a participatory, beneficiary-led approach, as well as its ability to capture and aggregate a number of benefits from a particular programme. Similar work is also taking place in Zambia focusing on programmes to reduce stigma. The piloting of SROI in a developing country context, (particularly rural India) is unprecedented, and as such the method had to be adapted, and a workshop component to be used with end beneficiaries developed. Overall despite a number of limitations to the method, which have been documented, concrete recommendations concerning the use of the SROI as a method for the Alliance, programme management, and policy implications are being considered.
Measurements across the Conceptual Framework In addition, the Alliance, New Economics Foundation and the Futures Institute have developed a funding proposal to support cost effectiveness assessments of community-based HIV interventionsviii. This includes a plan to develop a populated global database on the value for money of community HIV/AIDS interventions and a robust toolkit and capacity building service to allow civil society organisations to better generate cost and value for money data. The global value for money (VfM) registry/database of community based HIV/AIDS interventions, once populated, will include data on the (regional and nationally adjusted) costs of implementing the full typology of different community based interventions/activities as well as related social, environmental and economic outcome ‘value for money indices’ (and financial proxies). The development of a global ’value for money’ dataset will allow stakeholders to make comparisons and measurement of the relative costs of quality community based HIV/AIDS interventions as well as the social, environmental and economic value that a community response to HIV/AIDS results in, thereby enabling a more informed allocation of resources.
Issues There are a number of issues relating to value for money which have been raised as a result of work conducted to date and which are currently being discussed in the Alliance. These are briefly discussed here: •
What method should the Alliance use to track and monitor value for money of its programmes, approach and funding modalities? Currently, the Alliance is exploring a number of methods including tracking unit costs and social return on investment. The latter method would seem to offer the considerable attraction of being able to define and aggregate a range of outcomes accruing from a programme. However, the Alliance also needs to be able to measure and interpret unit costs because these are being widely used for value for money assessments. They probably do provide useful information on how to improve efficiency of funds and programmes provided that differences in services and their outcomes can be allowed for. Where possible, it would be useful to compare different methods in the same programmes.
Measuring and Improving the Value for Money of HIV Programming 6
DRAFT •
How can the broader Alliance family be engaged in initiatives to monitor value for money? It is important that this is not seen as a secretariat priority only but that it involves all linking organisations. For example, a key element of the success of the SROI work in India was the support provided to the process by Alliance India.
•
What capacity building needs exist within the Alliance for work on value-for- money? This is not an area where international development NGOs have a lot of experience. Materials that do exist may not be directly applicable to the areas in which the Alliance works. For example, in doing the SROI work in India, the Alliance was unable to find a good step-by-step guide that was relevant to the context. The Alliance adapted UK-focused material and conducted a half-day workshop for all those who would be involved.
• The Alliance will need to decide whether measuring value for money needs to be embedded within all its programmes and their routine reporting or whether this remains an element of special study or research. •
There are a number of risks that the Alliance faces in engaging with the value for money debate. There is a risk of the Alliance’s credibility being undermined if it is seen to be reliant on methods which are not seen as robust by stakeholders, particularly economists. This is a particular risk if it appears that the Alliance is only embracing methods which ‘make it look good’. Conversely, there is a risk of alienating other stakeholders, eg among the NGO community, who may be sceptical of seeking to ‘reduce’ all outcomes to economics. A particular challenge may be faced in the secretariat persuading linking organisations to embrace this approach, although there is good evidence from the Alliance’s experience to date that some linking organisations are very keen to explore this issue.
•
There is an underlying tension between the Alliance defining clearly its approach to measuring value for money and appearing to ‘put all its eggs in one basket’ by deciding on a particular method prematurely. The advantages of having a clear Alliance approach are many. It allows training materials to be developed and staff and others to be trained. It allows comparisons between programmes. The main risk is that a method may be selected which does not robustly and completely demonstrate the value for money of what the Alliance does.
• There are capacity issues for the Alliance to address if it is to focus more on issues of value for money. There may be a need to bring in more economics expertise and/or to link financial and M&E work more.
Next steps/future plans The Alliance will discuss the issues at the ‘Measuring and Improving the Value for Money of HIV/AIDS Programming of UK NGOs: Challenges and Opportunities’ meeting in December 2010, jointly hosted by the Alliance and the UK Consortium on AIDS and International Development. Topics for discussion include: • How UK NGOs might work together on costing HIV interventions. • Conducting efficiency studies including comparisons between different ways of providing the same services.
Measuring and Improving the Value for Money of HIV Programming 7
DRAFT • Linking up with the Global Fund on analysing their data. • Piloting and studying various approaches to cost effectiveness and sharing the results of those studies. • Linking with BOND value for money /effectiveness studies. • Sharing lessons with other sectors on value for money. • Linking with the private sector on measuring value for money. • How VfM approaches are already being planned into country programming.
Thanks to Roger Drew for coordinating the drafting of this paper. For follow up please contact Sam McPherson at the International HIV/AIDS Alliance smcpherson@aidsalliance.org
References i Alliance (2010) Evaluation of the Programme Partnership Arrangement between the Department for International Development (DFID) and the International HIV/AIDS Alliance 2008-2010 ii WHO, UNIADS and UNICEF (2010) Towards Universal Access: Scaling up Priority HIV/AIDS Interventions in the Health Sector: Progress Report 2010 iii DFID (2008) HIV/AIDS Factsheet iv Global Fund (2008) Value-for-Money in Global Fund-supported HIV, TB and Malaria Programmes: Framework and Proposed 2008-9 Workplan v Global Fund (2010) Improving Value for Money in Global Fund-supported Programs vi UNAIDS (2010) Technical Guidance Note: Addressing Value for Money in Round 10 Proposals for the Global Fund to Fight HIV/AIDS, Malaria and TB vii Mitchell, A. (2010) Making the Case for UK Aid available on http://www.dfid.gov.uk/Media-Room/ Speeches-and-articles/2010/Message-from-Andrew-Mitchell/ viii Alliance, NEF and the Futures Institute (2010) Improving the Value for Money of Community-Based Responses to HIV/AIDS ix
Marseille, E., Dandona, L., Marshall, N., Gaist, P., Bautista-Arredondo, S., Rollins, B., Bertozzi, S.M., Coovadia, J., Saba, J., Lioznov, D., DuPlessis, J-A., Krupitsky, E., Stanley, N., Over, M., Peryshkina, A., Kumar, S.G.P., Muyingo, S., Pitter, C., Lundberg, M. and Kahn, J.G. (2007) HIV Prevention Costs and Program Scale: data from the PANCEA Project in Five Low- and Middle-Income Countries BMC Health Serv Res. 2007; 7: 108
x Good practice HIV Programming Standards Alliance for use in the design, implementation and evaluation of Alliance programmes (2010) xi Alliance (2010) HIV and Healthy Communities: Strategy 2010 to 2012 xii NICE (2010) About NICE http://www.nice.org.uk/aboutnice/ xiii NICE (2007) Incorporating Health Economics in Guidelines and Assessing Resource Impact: Chapter 8 in the Guidance Manual xiv NICE (2010) NICE International available on http://www.nice.org.uk/aboutnice/niceinternational/ niceinternational.jsp xv Kabore, A. (2010) Costing Activities in Burkina Faso: A Case study from IPC Burkina Faso xvi Alliance (2010) Social Return on Investment: CHAHA Programme
International HIV/AIDS Alliance www.aidsalliance.org Cover image © Liza Tong for the Alliance
Measuring and Improving the Value for Money of HIV Programming 8