Evaluation: The Eastern Caribbean Community Action Project (EC-CAP) by International HIV/AIDS Alliance

Eastern Caribbean Community Action Project (EC-CAP): Evaluation Report By Jerker Edstrรถm and Paul Boyce

Revised 7th June 2011

USAID Cooperative Agreement No: 538-A-00-07-00100-00 This evaluation is made possible by the generous support of the American people through the United States Agency for International Development (USAID) in the Eastern Caribbean Region through the Eastern Caribbean Community Action Project led by the International HIV&AIDS Alliance and the Caribbean HIV & AIDS Alliance. The contents are the responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. For further information about this report, please contact the authors

Contents Acronyms ______________________________________________________ i Executive summary _____________________________________________ ii Acknowledgements ____________________________________________ viii 1. 2. 3. 4. a. i. ii. iii. b. i.

Introduction and background ________________________________ 1 Context, purpose, objectives and scope _______________________ 2 Methodology and ethical considerations _______________________ 3 Findings _________________________________________________ 6 Use of strategic information to inform interventions and design ............... 6 Use of strategic information in programme design ................................... 6 Role and value added of the partnership with UCSF ............................... 9 Influence of SI in contributing to national responses .............................. 10 Improving the access of key populations to HIV community services .... 11 Changes in knowledge of HIV risk reduction achieved and contribution of strategic information activities .................................................................. 11 ii. Beneficiaries’ perceptions of changes in risk behaviours ....................... 14 iii. Beneficiaries’ perceptions of levels and quality of HIV service access .. 16 iv. Results and effectiveness of community-based rapid HIV testing in Antigua and Barbados ........................................................................... 19 v. Role and value added of the partnership with Intrahealth ...................... 21 vi. Effectiveness of bi-directional referral systems, Antigua and Barbados 23 c. Capacity building and civil society involvement ..................................... 25 i. Strengths, challenges and responsiveness of CBO capacity building approaches .............................................................................................. 25 ii. Contribution of the grants programmes to results .................................. 27 iii. Relevance of the approach in the Caribbean context ............................ 30 iv. Efficiency and sustainability of activities and achievements .................. 32 v. Effectiveness and relevance of management for objectives .................. 33 5. Conclusions and lessons learned ___________________________ 36 a. Conclusions on use of research, M&E and strategic information (SI) .... 36 b. Conclusions on CHAA’s improving access by MARPs to HIV services . 36 c. Conclusions on CHAA’s operational approach to civil society strengthening and its associated business model .................................... 37 6. Recommendations ________________________________________ 38 a. Recommendations for USAID ................................................................ 38 b. Recommendations for the Caribbean HIV&AIDS Alliance ..................... 40 c. Recommendations for the International Alliance .................................... 42 Annexes ______________________________________________________ 43

Acronyms CA CBRCT CHAA CRN+ EC-CAP GFATM IDS IHAA M&E MARP MARPs MSM NAC NAPS NAS PAHO PANCAP PLHIV PO RCT SI SISTA SW/s UCSF USAID UWI VCT

= Community Animator = Community Based Rapid Counselling and Testing = Caribbean HIV&AIDS Alliance = Caribbean Network of People Living with HIV = Eastern Caribbean Community Action Project = Global Fund to Fight AIDS, Tuberculosis and Malaria = Institute of Development Studies = International HIV/AIDS Alliance = Monitoring and evaluation = Most at Risk Population (typically MSM, SWs or PLHIV) = informal for individuals (in plural) thought to belong to a MARP = Men who have sex with men = National AIDS Commission (as in Barbados) = National AIDS Programme Secretariat (as in Antigua & Barbuda) = National AIDS Secretariat (as in St Vincent & the Grenadines) = Pan-American Health Organisation (Regional Office of WHO) = Pan Caribbean Partnership Against HIV and AIDS = Person/s Living with HIV = Programme Officer = Rapid Counselling and Testing = Strategic Information = Sisters Informing Sisters About Topics on AIDS = Sex Worker/s = University of California in San Francisco = United States Agency for International Development = University of the West Indies = Voluntary Counselling and Testing

Executive summary The Eastern Caribbean Community Action Project (EC-CAP) aims to increase access to HIV and AIDS community services for most-at-risk populations (MARPs) through evidence based programming. It is funded by USAID (USD$10.4 million over 3 years - ending February 2011), through the International HIV/AIDS Alliance (IHAA) and its Linking Organisation the Caribbean HIV&AIDS Alliance (CHAA), to work in Antigua and Barbuda, Barbados, St Kitts and Nevis and St Vincent and the Grenadines. The aim of the Project is to increase access to HIV and AIDS services for most at risk populations through evidenced-based programming. The expected results of this programme are (1) increased use of strategic information to promote sustainable evidence-based HIV/AIDS community services and (2) increased access to community services. As the current project is at the end of its three year contract cycle, IHAA called for an external team to undertake a programme evaluation, then selected and recruited the Institute of Development Studies (IDS). This evaluation was lead for IDS by Jerker Edström working in close collaboration with Dr Paul Boyce, the authors of this report. The methodology for this evaluation was guided directly and pragmatically by the specific evaluation questions, provided by the International HIV/AIDS Alliance, which demanded a range of techniques, as well as a strategic range of informants and sources of information. The work was divided into three phases, which included a preparatory phase prior to country visits, followed by two phases in the region. The key methods employed included: (i) Structured interviews with a broad range of key informants; (ii) Semi-structured one-to-one interviews with ca. 10 service clients in each country; (iii) A rapid service satisfaction survey of 148 outreach clients and (iv) occasional loosely structured discussions in small groups. Summary of findings and conclusions In terms of the programme’s ‘use of strategic information’, we found that, indeed, EC-CAP is evidence based in its design and in many more specific locally tailored aspects – that is based on internationally established good practice, available regional evidence and local information. This is particularly so if we recognise the wider range of types of information included under SI, beyond formal research – including the contextually nuanced feedback of animators as it guides day-to-day programme implementation (which, for example, is sometimes entered on log-sheets but underutilised in reporting or analysis). However, most research studies under the project have come to fruition at the end of the three year funding cycle, meaning that findings have not been available to fully inform programme development. Lack of sufficient inception period compounded this problem – many SI studies were not properly started until at least a year into programme roll-out. The time lost due to set-up and staff ii

turn-over had a major impact on the development of SI research and M&E systems – disrupting the development of coherent means of feedback and consistent lines of management within M&E components of work. The research partnership with Intrahealth, which was focused on getting a particular job done over time, appears to have been fairly successful and perhaps of more practical value to the work on the ground than the research partnership with UCSF. With some exceptions, the latter did not necessarily add substantially to contextual knowledge of social and cultural settings, although some of this may have been down to complicating factors associated with the ambitious design along with set up and turn-over challenges. CHAA could now usefully engage a more diverse range of research and SI partners, including and especially from within the region. Research could be more practical and integrated better with M&E and organisational learning. In particular, M&E in country captures a lot of important psycho-social and contextual detail pertaining to the life experiences, vulnerabilities and risks of the key populations with whom animators work. To date this important data has neither been adequately utilised in M&E systems, nor as a source of research in itself. Our findings on ‘improving access by MARPs to HIV and AIDS services’ can be summarised as follows. Access has radically improved for MARPs in the programme countries covered by EC-CAP and CHAA has played a unique and pivotal role in developing and enhancing this. Work carried out to date has begun to institutionalise the idea that access for MARPs must be a key component of national HIV/AIDS strategies. This is notable in each of the countries where CHAA is operating. Whilst significant steps have been taken integration of MARPs-oriented activities are nonetheless nascent within national strategies, but this is not sufficiently institutionalised as yet to develop further in the absence of further capacity building and agenda development through ECCAP. So far, the outreach has mainly provided information, support and supplies, but may need to be complemented with strategies for building social capital and addressing ‘structural obstacles’ (e.g. to accessing services but also to behavioural adaptations) for MARPs more explicitly and strategically (but based on local realities). CHAA’s operational approach to ‘civil society strengthening and its business model’ for the work can be summarised as follows. Among both state and nonstate stakeholders in all four operational countries, CHAA is highly valued for its engagement and support of communities and organisations – it plays a unique role in this regard – as well as with government structures. It is important to recognise the challenges in strengthening civil society in small Eastern Caribbean countries. CBOs addressing the needs of MARPs are small and of limited organizational capacity in all countries where CHAA operates. iii

The strengthening of fledgling support groups for MARPs has been less successful than engaging with the few more established groups. It is clear that a long-term and more intensive strategy will be needed to support groups for MARPs – including significant support in organizational capacity development and means of securing funding beyond CHAA’s onward granting. Close ‘handholding’ relationships with CSOs, MARP clients and national stakeholders is CHAA’s particular strength on the ground, but this is thinly stretched and not fully meeting its potentials. Organisational management structures and approaches, whilst apparently streamlined – eg. by sharing functions across islands – are nevertheless ‘top heavy’ and senior staff turnover makes this approach and structure all the more challenging. Whilst clearly having been effective in setting up the programme and supporting the development of the work in the first few years, the value added of IHAA to CHAA appears to have become less clear and less visible on the ground in recent years, with a concomitant need to revisit the role of the IHAA secretariat in relation to CHAA and that of the latter in relation to the broader ‘Alliance family’.

Recommendations Recommendations are provided separately for USAID, CHAA and IHAA, below.

Recommendations for USAID:  USAID should invest in, utilise and expand the programmes built up in ECCAP, but: (i) over a longer time-frame and (ii) with increasing levels of resources going into country-based activities.  USAID should broaden the coverage in the sub-region with these or similar programmes, paying particular attention to patterns of mobility in most-at-risk populations.  USAID should reconsider updating its approach to promoting research in the region, by focusing on building relevant research learning capacities and partnerships within the region and with more limited and more strategic inputs from northern research institutions.  Create an environment where research can be better informed by the needs of practitioners for delivering more effective services and where USAID’s lead partner/s in the region can draw more flexibly on a range of regional and global technical support providers and partners.

 Strengthen the complementary role of qualitative and social science research aspects to strengthen work on marginalisation, sexuality and gender for example – especially as such research improves and broadens the evidence base for psycho-socially relevant and contextually effective HIV prevention.  Enhance a research approach that is not only better regionally located but also pays better attention to heterogeneity within the region – focusing on the needs and social contexts of different countries as much as the Caribbean overall.  USAID should continue support NGOs, CBOs and FBOs in the response to HIV and AIDS, because of their complementary roles which make programmes well rounded, appropriate and sensitive to local contexts.  USAID should scale-up the model to more countries, and strengthen possibilities for linkages between countries and between country and civil society movements and processes at regional levels.

Recommendations for the Caribbean HIV&AIDS Alliance:  In strategic information and strategic communication, focus on improving monitoring, documenting and representing the content of what CHAA does best – i.e. good community driven animator work, making more use of CHAA’s wealth of – and access to – rich qualitative information.  Orient research activities towards more operations research linked more strongly to M&E, drawing more strategically on a broader range or research partners from within the Caribbean and beyond, whilst facilitating and maximising learning across programme countries.  Recognise the increasing capacity of the community animator’s and their potential as researchers within facilitated participatory and communitybased research designs.  Develop longer-term strategies for linking research and learning to programmatic development and improvement, with greater attention to timeliness and synchronising outputs with programme requirements.  Build on and expand the excellent community animator programme, in four ways: (i) into new and nearby countries; (ii) with increasing levels of skills in counselling and behaviour change communications;

(iii) linking the work of animators more strongly to community advocacy and local CBO support group strengthening (e.g. PLHIV, MSM or lowincome women’s groups) – such that skills development of animators is better linked to and institutionalised with capacity building for CBOs (iv) strengthen connections with similar programmes in larger countries in the broader region, incl. for regional referral and networking support (esp. in Jamaica, Dominican Republic and Guyana)  Invest seriously in developing the NGO/CBO support function at country level, by: (i) Starting with organic flexibility, but guided by strategic long-term goals, further develop systematic approaches to gradual capacity strengthening of particular high priority CBOs (ii) Extend the rationale for these partnerships to more functions beyond service delivery to include documentation, campaigning and/or advocacy – with in-country and regional training accordingly (iii) Maintain and further develop a ‘civil society sector-wide’ approach to a strategic range of partnerships, from the intensive and focused ‘handholding’ relationships to the more established NGOs and FBOs (iv) Improved and more structured (but simpler formats for) application and support procedures – e.g. include simple and clear guiding questions for applicants to address. Such procedures should also include and facilitate more rapid turn-around with processes and grant payments.  Protect existing local programmes and build these up further, by strengthening staffing and support at country level, empowering country office staff further, with more down-streamed decision-making abilities.  At all cost, avoid stretching country level staff thinner geographically.  Connectedly, rationalise regional level functions, within long term goals to (i) have more staff from within the region (ii) more staff from each country within programme offices (in addition to animators) (iii) rebalance the staffing pyramid with a stronger base and leaner apex.  Invest further in regionalising CHAA’s image and connectedness, e.g.: (i) by continuing to reach out to critical regional stakeholders and power brokers and proactively ‘inviting them in’ (including on the Board) (ii) enhancing country-based programme office structures through improved country-focused recruitment (iii) consider continued re-branding, but doing so with local and regional engagement in the process and outcomes

Recommendations for the International HIV/AIDS Alliance:  IHAA needs to engage more closely with CHAA to find out what it’s most useful role can be in support of CHAA - both in relation to the USAID bid as well as beyond.  IHAA will need to respect CHAA’s decisions about its own directions, but think creatively about what it (IHAA) can offer CHAA, be it in relation to broader organisational development or as a key player within the broader Alliance family and across the region (e.g. How should CHAA relate to Haiti?).  What role might IHAA play in developing and enhancing work with MARPs which has been CHAA’s ‘signature achievement’ – is there a role in facilitating linkages and learning in work with key populations in the Caribbean with the work on other IHAA LOs?

vii

Acknowledgements This evaluation was led by Jerker Edström, of the Institute of Development Studies (IDS) in the UK, with the close collaboration of Paul Boyce, from the Institute of Education (IoE), and supported by a wide team of CHAA Programme Officers and Animators (or, outreach workers) as members of the evaluation effort, as well as in numerous ways administratively and practically by a broader group of colleagues in the various offices of the Caribbean HIV&AIDS Alliance and the International HIV/AIDS Alliance. We are indebted to Anamaria Bejar and Liza Tong (of IHAA) and Mr Basil Williams and Jane Armstrong (at CHAA) for their guidance on how to approach the evaluation, as well as for their respective substantive perspectives as key informants, and for feed-back on the draft evaluation report. At country level, we would particularly like to thank each CHAA programme officer – Teddy Leon, Louise Tillotson, Kevin Farara and Nadine Kassie - and a number of others listed in the key informants listed in Annex 2, below. We are also deeply grateful for the support of the extended team of CHAA Animators and community volunteers who engaged in the evaluation as front-line workers on the client satisfaction survey and in-depth client interviews, again listed in Annex 2. Particular thanks for excellent logistical support go to Natasha Ward, Sharla Elcock, Avelyn Gittens and Kevin Ramjohn (of CHAA), Karine Gatellier and Marieke Deville (at IHAA), as well as Jan Boyes (at IDS). In addition, we are very grateful for the support of in translation and data entry of client satisfaction survey data by two students Maria Berbegal Ibáñez, at IDS, on a Research Assistant basis and of Matz Edström, at City University (London), on a volunteer basis. We thank all the numerous contacts and stakeholders provided essential information and perspectives as key informants, and who are listed in Annex 2. Crucially, we would like to acknowledge and thank the ca. 150 (here unnamed) community members from most at risk populations, who gracefully donated their time and cooperation in the evaluation’s client satisfaction survey, as well as in qualitative in-depth interviews. Last but not least, our heartfelt thanks go out to the core-team of CHAA animators and POs who contributed to the evaluation methodology and design and showed admirable professionalism, sensitivity and perseverance throughout the work. Whilst all have provided essential inputs and views, we take full responsibility for the conclusions, views and any unintentional errors presented in this report.

Core-team at methodology development and training workshop. Clockwise from left: Carlisle, Adele, Donovan, Amanda, Teddy (PO), De, Lydia and Kezreen (names in Annex 2).

viii

Eastern Caribbean Community Action Project (EC-CAP): Evaluation Report 1. Introduction and background The Eastern Caribbean Community Action Project (EC-CAP) aims to increase access to HIV and AIDS community services for most-at-risk populations (MARPs) through evidence based programming. It is funded by USAID (USD$10.4 million over 3 years - ending Sept 2010), through the International HIV/AIDS Alliance (IHAA) and its Linking Organisation the Caribbean HIV&AIDS Alliance (CHAA), to work in Antigua and Barbuda, Barbados, St Kitts and Nevis and St Vincent and the Grenadines. EC-CAP is based on the premise that access to HIV services for MARPs can be achieved through evidence-based programming, community and civil society involvement, stronger engagement with national programmes and enhanced behaviour change interventions. The aim of the Project is to increase access to HIV and AIDS services for most at risk populations through evidenced-based programming. The expected results of this programme are (1) increased use of strategic information to promote sustainable evidence-based HIV/AIDS community services and (2) increased access to community services. Six complementary approaches drive the programme toward achieving these intended results, namely:  Community outreach delivering evidence-based interventions  Prevention and care efforts focusing on risk behavior rather than risk groups  Development and analysis of strategic information (SI) to inform and guide approaches to HIV/AIDS programming  Addressing stigma and discrimination as a priority to service delivery  Active participation of beneficiaries and local partners  Investments in civil society organizations through small grants. Central to EC-CAP is a peer-led approach, utilising Community Animator outreach workers, drawn from the target populations. One-on-one interventions with those most-at-risk of exposure to HIV remain one of the main approaches of supporting effective behaviour change. Animators through their direct links at community level are best placed to access peers. As the current project is at the end of its three year contract cycle, IHAA called for an external team to undertake a programme evaluation, then selected and recruited the Institute of Development Studies (IDS). This evaluation was lead for IDS by Jerker Edström working in close collaboration with Dr Paul Boyce, the authors of this report. In this evaluation report we present the aims, objectives and findings of the evaluation, as well as our recommendations to IHAA, CHAA, regional stakeholders and USAID. 1

2. Context, purpose, objectives and scope USAID provided a cooperative agreement to IHAA for the programme and the programme is implemented in the field by CHAA, whose main country office is based in Port of Spain, Trinidad. At the time of the agreement, CHAA was a regional office of IHAA. In December 2008 CHAA became an independent linking organisation, which is a member of the International HIV/AIDS Alliance. Two additional international partners have contributed to key aspects of the programme: University of California in San Francisco (UCSF) had a set of key deliverables related to the strategic information and research component of the project. Intrahealth, based in Georgia, USA, was responsible for deliverables related to the Community Based Rapid Counselling and Testing (CBRCT). The programme was piloted in Antigua and Barbuda, Barbados and onward grants are provided to community based organisations (CBOs) in all countries to implement aspects of the work. As specified in the Terms of Reference (TOR) developed by IHAA, the main aim of this evaluation is to assess the achievements of EC-CAP from November 2007 to September 2010, particularly the extent to which the programme has increased access of most at risk populations to HIV services and to what extent strategic information successfully informed evidence based design of interventions. It is hoped that this evaluation may contribute to improving the design of future programmes in the Caribbean, which involve groups particularly at risk of HIV infection. More specifically, the evaluation is being undertaken to provide useful lessons for a new phase of the programme and to meet the USAID requirement for an end of programme evaluation. The intended audiences of this evaluation include CHAA, IHAA, USAID, regional and national Governmental stakeholders, engaged activists in the target populations, project partners, local CBOs, community level stakeholders, and other international organisations. The specific objectives of the evaluation are three-fold and include the following: 1. To assess the project’s achievements in developing and analysing strategic information to promote evidence based design of HIV/AIDS community services. 2. To determine the project’s contribution to improving the access of MARPs to HIV community services. 3. To assess the programme’s achievements, lessons learned, effectiveness and relevance, focusing particularly on the role of community (and community animator outreach workers) and civil society involvement in the HIV/AIDS response The chapter on evaluation findings, below, is structured into three corresponding sections, each of which contains further sub-sections focusing on more specific evaluation questions outlined in the evaluation TOR (Annex 1).

3. Methodology and ethical considerations The methodology for this evaluation was guided directly and pragmatically by the specific evaluation questions, which demand a range of techniques, as well as a strategic range of informants and sources of information. Privileging the clients’ or relevant community members’ perspective is the starting point and the method aims to place this in historical context, relating communities’ access and experience to the institutional historical context of the programme within the broader Eastern Caribbean landscape. The methodology:  Is pragmatically driven by the complex research questions (complex insofar as combining bottom-up user views with those of programmers and providers in diverse contexts);  Triangulates direct views of key informants in the programme with other sources of information;  Combines descriptive quantitative data with more analytical qualitative perspectives and;  Engages members from the communities served, or intended programme beneficiaries, in direct ways and in ways which take due account of ethics of the evaluation. The work was divided into three phases. This included a preparatory phase prior to country visits, followed by two phases in the region: One with detailed assessment and planning work with CHAA and key informant interviews at the regional level in Trinidad and Barbados, as well as for a training of a core group of ‘volunteer’ evaluation team members from – or strongly linked into – client communities (such as sex workers, MSM and PLHIV, or ‘MARPs’), most of whom were also peer animators in the project; The second visit consisted of a number of country visits, where the two UK based consultants divided up for part of the time to work in a more intensive and in-depth evaluation weeks, covering the four different countries. Phase one was carried out in Brighton, UK to structure and plan the assignment in detail, in close collaboration with colleagues at IHAA and in telephone discussions and e-mail with the staff of CHAA. In this phase we gathered relevant programme and project information and data, along with general reading of complementary research from the region through academic and other outlets. We scanned published and grey literature including: IHAA documents on both the region programmes and MARPs; comparative reports and literature on MARPs and related interventions. Comparative literature was used in the evaluation research to better assess, understand and contextualize the situation in the Caribbean, with specific reference to interventions with most at risk groups. Finally, the overall methodology for in-country missions was developed in detail and in consultation with IHAA and CHAA. The second phase provided the overview of the programme, its institutional and historical context, as well as allowed the Team to prepare a core group of volunteer animators to take part in the evaluation process for the remainder of 3

the assignment. We specified desirable criteria for the evaluation team members and colleagues at CHAA facilitated the selection three volunteer animators from each programme country. One from each of St Kitts and from St Vincent, as well as two animators from Antigua, attended an initial planning and training workshop in Barbados, along with participation of three animators from Barbados and the CHAA Programme Officer of the Barbados office. Methods and tools were refined in this workshop and the consultants and participants undertook a number of interviews together with sex workers and men who have sex with men to test the tools, and as part of the evaluation itself. In addition to methods development and animator training and field research in Barbados, phase two also focused on interviews with a range of regional key informants, including in Barbados (notably the USAID Cognisant Technical Officer and CHAA staff with regional support roles in the project) and at CHAA’s regional headquarters in Trinidad, where CHAA regional staff were interviewed, as well as regional stakeholders both in person or via telephone interviews. Phase three focused on field work in the three remaining programme countries, starting with a short planning workshop/meeting in each, where the UK consultant/s, the CHAA’s in-country PO and the selected volunteer animators planned 4 days of intensive evaluation work, including; for conducting some oneto-one structured interviews with members of the most at risk populations together, conducting key informant interviews with national stakeholders and implementing partners, collecting data from service clients with a concise ‘access and satisfaction questionnaire’ and a range of protocols and guidelines for discussions and interviews. Whilst the initial intention had been to include some peer-led participatory group analysis, it became clear that the time available, skill base and context (where group meetings were hard to organise, in for example strip-clubs or on the streets at night) did not allow for more in-depth participatory assessments in groups. On the other hand, animators were closely involved in conducting interviews and CHAA Programme Officers partook in some interviews as well as acted as key informants themselves. Whilst the client access and satisfaction survey reached a large number of clients (148 completed questionnaires) and generated a wealth of quantitative data, as well as some qualitative explanations/elaborations, results need to be treated with caution, due to: (i) the inherent biases in the sampling method (being those clients the animators and volunteers could reach within the time-frame), (ii) the data collectors being – in the majority of cases – animators themselves, and (iii) the potential that the evaluation could be seen as a sign of the programme coming to an end. Hence, we can expect overly positive replies, but this was somewhat mitigated by a range of factors, such as (a) the animators good understanding and communications capacity to explain the purpose and process to their peers, and (b) the UK evaluators accompanying animators on a number of occasions to observe interviews, as well as (c) complementary, more in-depth, one-to-one interviews being undertaken with a smaller number of clients, for a 4

fuller exploration of the issues at stake. On balance, whilst likely overly positive, we feel that answers were useful and informants answering broadly truthfully, and that the information gathered is in line with accounts from other keyinformants, including government stakeholders. With this range of methods, we explored and document different ideas of 'effectiveness and relevance' of MARPs specific programmes. We also took account of timelines and scales over which effectiveness and relevance might be conceived and measured - long term effectiveness in terms of impact on HIV methodology, for example, may not be obvious for some time. We explored the challenges that this presented in terms of demonstrations of programme effectiveness for CBOs, CHAA and IHAA, especially in dialogue with relevant health ministries and other key informants. We also explored the complexities of behaviour change, which cannot all be explored in respect of a 'hypodermic model' (i.e. a model whereby simply ‘injecting’ HIV prevention information is seen as resulting in behaviour change or other outcomes) but must necessarily be understood as multi-dimensional and complex in peoples’ lives. This required taking context more fully into account, and the peer-oriented community-based aspects of the methodology allowed us to explore these issues in fairly nuanced detail. For example, we examined ways in which the work of successful community animators is not simply premised on health education but in developing relationships of trust and care that take account of the total ‘lifeworld’ of vulnerable beneficiaries. We also explored what it means to ‘be a MARP’, as it were, and examined ways in which intervention approaches focused focused on risk behaviour and MARPs may be helpful in addressing stigma, whilst we also explored whether beneficiaries defined as most at risk may be stigmatised in new ways. This was particularly important in terms of exploring ownership of health messages and involvement community HIV services, but also around the meaningful involvement in programme structures, policy influence and decision making itself. Both consultants worked together in collaboration with the local volunteer team members in the field to analyse interviews and group discussions according to the key research questions as outlined in the terms of reference. The in-country POs of CHAA (and other staff) supported the teams proactively and with great insight and integrity, without in any away appearing to try to influence the overall effort. Whilst the great range of stakeholders and informant were interviewed in the region during the field work, the team interviewed some contacts (in Canada, USA and Guyana via telephone or Skype). The fuller synthesis of findings from the countries and programmes was compiled and elaborated by the team following the field visits and then shared with IHAA and senior management at CHAA for comment. The final synthesis is reported below.

4. Findings The findings of the evaluation are presented in three main sections relating to the three objectives of the evaluation, with sub-sections addressing each specific evaluation question specified by IHAA. At the outset, however, it is worth pointing out that – overall – the Alliance’s USAID supported programme EC-CAP is broadly seen at the regional level, as well as in programme countries, as very significant: That is, significant in terms of addressing the issue of HIV transmission amongst most at risk populations in the Eastern Caribbean. It was almost universally acknowledged that this area of programming has been slow to develop in the sub-region and the difference brought by the Alliance/CHAA is generally recognised and welcomed, by National AIDS Programmes and other stakeholders in the response. One key stakeholder in the region felt that: “… [CHAA] has enhanced national programmes and CHAA is seen as a key partner in programmes at national and regional levels. National AIDS Programmes take them very seriously, but governments more broadly… that depends and it is harder to say, as HIV is pretty far down the list of priorities. However, at PANCAP level, for example, CHAA’s work is taken very seriously.” (Ian McKnight, Caribbean Vulnerable Communities, CVC) This recognition is especially strong in programme countries, whilst in terms of certain stakeholders in the broader Caribbean region CHAA’s role and contribution is less evenly appreciated. For example CHAA’s Executive Director, Mr Basil Williams, explained that in relation to important regional stakeholders in the Caribbean you have to “push the model and market your approach without looking as if you are”. He also felt that CHAA is slowly becoming more accepted. The main problem has been that CHAA has not been seen as an indigenous organisation – there has been a lot of work to present CHAA as regional, developing a Board with regional trustees and so forth.

a. Use of strategic information to inform interventions and design i. Use of strategic information in programme design First, the understanding of the term and notion ‘strategic information’ (SI) was variable across informants – ranging from nuanced understandings of multiple types of information use, on the one hand, to SI being seen specifically as scientific research projects or formal studies, on the other. The latter view sees it as distinct from monitoring and evaluation, or less formal assessments or ‘intelligence’ gathering. Whilst understandings do vary, CHAA’s emphasis on this type of work more broadly, is seen by many has having been very positive, contributing to a stronger culture of evidence-based programming. 6

“The area of SI has been a challenge in the region. We’ve had some SIs but not many M&E frameworks. It’s only recently that SI has become a priority – especially in the last three years. Recent CHAA projects… have generated information for more concrete action across the Caribbean region. I would say that the Alliance has done this more than others.” (Carl Browne, Chair of PANCAP and Board Member of CHAA) The vision of the grant (from USAID) was said to take an evidence-based approach, whilst some felt that this has not happened quite as envisioned, at least if one sees the commissioned SI studies as the basis for programmes. These have had little programmatic influence – partly because of the sequence and timing of reports production, but many also questioned the relevance of certain studies for practical programme recommendations. This may, in part, reflect a fairly restrictive paradigm of research, whereby independent research uncovers facts to then be addressed by programmes (which is indeed difficult to achieve with parallel timelines), as well as the fact that priorities for research could not be easily informed by programmes at the outset before experience was gathered, as it were. If research was to be the basis for programmes, the identification of priorities for research may in fact have been premature, at least in some cases. It is possible that adaptations to the research process might have allowed for greater engagement with programme priorities and realities, or that a more fundamental re-think about how to make information generation more strategic (with a more open-ended menu of methodological and research paradigmatic options) is called for. The most salient example of where an SI study has directly fed into programmes has been in St Kitts, where the UCSF led study on lower-income women working in factories to assess the potentials for adapting the SISTA model developed and tested in the US. This was seen as particularly relevant because of common patterns of concurrent sexual partnerships amongst low-income women in the region and CHAA has developed a factory based peer-education programme on the basis of the findings. Other work of this kind, such as a study amongst men who have sex with men in Barbados, similarly assessed the relevance of the USdeveloped model “MPowerment”, but this came too late in the cycle for developing a project. The key question raised by many in relation to some of these studies was one about their actual usefulness in terms of developing contextually relevant and detailed strategies beyond the basic question as to whether a tested concept from another country may or may not be applicable. Whilst some also pointed out that conclusions from, for example, the FBO studies were often very obvious and questioned the value of “researching common sense”, many others nonetheless felt that it was useful to have such basic conclusions formally documented for influencing policy, if not programme design, even if we did not come across concrete examples of such policy influence.

Some felt that, as the programme became more localised and CHAA developed its use of local studies and information in the design of strategies, they were not guided exclusively by what was ‘given to them’ from abroad. This was said to be good as this has allowed the organisation to ventured into more locally specific groups and issues, such as MSM selling sex or – like in Antigua – progressing from street based sex workers, to clubs to issues of women and poverty more broadly. A key problem with much evidence at a global or even regional level is that it is not specific enough to become truly useful. For example, CHAA’s Evaluation Director, Caroline Allen, reflected that: “There is a lot of research from the region more generally on partner concurrency of low income young women in the Caribbean, so that part has some evidence. There is a lot of talk of a ‘feminisation of the epidemic in the Caribbean’, but much of that is due to data from a few larger countries like Haiti and it is not really true of the Eastern Caribbean in general, so there is also a need to look much more at young men with multiple partners or multiple sexual encounters.” Beyond formal research studies, the use of information and data gathered in the day-to-day programme development and management process (such as monitoring data, animators’ and clients’ feed-back and other sources) feeds into both into adapting their own strategies and informing National AIDS Programmes in all the programme countries. As a National AIDS Programme Director noted; “Before the Alliance there was no information on MARPS, but the Alliance and its’ animators had information that we didn’t have. Prostitution and buggery is illegal, so we could also have been seen as condoning it if we were going into that area. But CHAA could identify hotspots and work with the groups.” Quarterly reports are sent to the NAP in each country, which is seen as a fundamental contribution. Beyond this basic function, or the one of keeping track and reporting results to the donor, there is a wealth of useful but underutilised strategic information within CHAA’s systems. For example, the animators’ log sheets record risk behaviours – with some space for narrative feedback – but this is not used to its full potential. One CHAA in-country Programme Officer observed that “contextual, anecdotal detail is not being captured in the current M&E work, which actually misses a lot of what CHAA is about and really good at – the work that the animators do, good community-based MARPs specific stuff. There is no real system for capturing this.” Overall CHAA has made considerable progress and some quite fundamental contributions to evidence-oriented programming in the region. In the future, CHAA would do well to develop its use of strategic information in ways that better include, value and utilise the wealth of rich contextual information and data that is being generated thought the work of the animators and programme officers. That would entail developing the organisations thinking and approach to ‘what counts as evidence’ and how to use it more strategically. 8

ii. Role and value added of the partnership with UCSF The programme bid included the University of California, San Francisco (UCSF) as a partner in order to strengthen research capacity and in view of USAID’s stipulation on the need to have international collaborators. Whilst several informants felt that UCSF brought credibility and strengthened CHAA’s capacity in research (and facilitated getting ethical research approvals), there were mixed feelings as to their overall contribution to programme impacts in the end. Many informants – including the Principal Investigator at UCSF, Janet Meyers – felt the relationship has helped shift the culture of CHAA and national partners some way towards a stronger emphasis on evidence-based programming. On the other hand, various reservations were expressed around (a) the timing and timeliness of studies conducted and disseminated, (b) the quality and usefulness of some of the studies (beyond in terms of timing) and (c) certain issues around poor communications or collegiate relations. Broadly, UCSF’s research and reports were planned to have been completed earlier in the roll-out of CHAA’s work, but in practice that has not happened. First, three years was not felt to have been a viable time-frame to develop ‘SI’ research studies and associated reports, as well as to then develop strategies based on the findings. CHAA was still finalising SI reports at the end of 3 years. Significant time was lost in setting up the programme structure, compounded by staff turnover in the research and M&E teams at CHAA. Such delays were said to also be due to several stages of review at multiple levels and re-editing by senior staff at CHAA and IHAA before sign off, so were not due to UCSF primarily. The Executive Director recalled how “some of the output from UCSF has been very disappointing, with weak content and basic errors of definition.” For example, a few respondents commented on an issue where UCSF wanted to extensively use the slang term “batty-man” in the MPowerment report. Several other CHAA staff also objected that this and similar language would seem very offensive in such a report and that it would discredit the report in the eyes of others. There was a broadly felt lack of understanding of these regional sensitivities in UCSF’s report writing style. Beyond ‘sensitivities’, several CHAA staff also believe that UCSF did not understand the full context of Caribbean culture and that, consequently, the partner made basic and substantive errors. For example, field work carried out by UCSF was said to typically be short-term, such as two weeks in St Vincent. Consequently, the data and analysis was felt to be poorly contextualised and ‘thin’. Ms Meyers of UCSF acknowledged that there had been “a bit of a culture clash” and that UCSF was new to the region, but that “we learned a lot”. These and other issues to do with quality of writing and basic contextual misinterpretations was also said to have soured the research relationship. Senior staff felt that there was some arrogance on the part of UCSF and an intransigent attitude towards feed-back. However, as more experienced staff came into CHAA, the organisation gained more research confidence allowing 9

them to ‘push back’ more effectively on some of the wanting analysis that UCSF produced. Overall, whilst useful for initial credibility, the partnership with UCSF was probably not a particularly useful or successful collaboration, on balance. For the future, research may need to become more operationally focused, based in the region and better integrated with CHAA’s own monitoring and learning.

iii. Influence of SI in contributing to national responses On reflecting on whether CHAA’s research and SI activities have influenced national responses, the Regional Director of UNAIDS in Trinidad usefully framed a key issue also posed by some others: “CHAA can work with MARPs and, so, the NAPs don’t have to… Is this influence or just taking on an agenda without the necessarily influencing? There is good pragmatic influence, but no policy influence per se.” At one level, this conclusion has merit, since many government representatives openly acknowledge their challenges in working with communities most at risk, as also expressed by the Director of NAPS in Antigua (see p. 8 above). Nonetheless, other felts that there is emerging strategic and real influence by CHAA within national policy environments in diffuse ways. How this happens varies somewhat by country and in Barbados, for example, the Assistant Director of the National AIDS Council, Nicole Drake, explained that “the current – 2008 to 2013 – plan was already developed and focused on MARPS, but CHAA gave us more specific direction, as they have a good comparative advantage in this area.” Similarly, as a new strategic plan is about to be developed in Antigua, it was said that CHAA’s influence has strongly put MARPs on the agenda. In St Vincent and the Grenadines, Rose Clair-Charles, a VCT clinic peer, explained: “…the way the programme is designed, people feel more comfortable coming to the Alliance than to the National Programme – especially the marginalised. Because of the Alliance’s openness, I can see – coming back now – I can see a big change (I left in ’97 and came back in 2008): As a result of the programme these people had ‘a voice’, so to speak” There appears to have been a significant influence on rapid testing, as a strategic information-based programme in Antigua and to some extent Barbados. One informant explained that “… counselling and testing in Barbados and Antigua, this has been taken forward because of CHAA involvement. Similarly the MoH in Barbados has made mobile testing available.” In some countries, CHAA has been involved in consultations and working groups for developing the national plans and strategies. For example, a UCSF Director and CHAA staff engaged with and sat on working groups for M&E in the plan development in St Vincent, as described by Margarita Tash, of the main Infectious Disease Clinic: “They were part of it when the secretariat did their strategic plan… Alliance would report to them. I know they are involved… on the 10

monitoring and evaluation working group for the M&E plan. They were also in the Strategic Plan consultation. They did an assessment of most at risk populations – I have it – I think they did put something in the Strategic Plan to work with most at risk groups as a result. We are aware that they work with MARPs. Similarly in St Kitts people working within the national AIDS programme considered CHAA to be an intrinsic facet of national programme roll-out, especially in respect of the involvement of MARPs, which it was reported would be unlikely to take place without CHAA, at least at the level at which it is currently developing. In addition, UCSF explained that plans by Gender Affairs were directly informed by the data and information emerging from the SISTA study. It is worth pointing out that the nature of the actual influence wielded by CHAA, or others, on National Programmes and strategies take many forms and through engaging at many levels. Also, working with FBOs on stigma and discrimination in the congregations and communities is likely to have subtle influences over time. Although the motivation behind carrying out the FBO studies was felt by our UCSF informant to have been likely political, rather than based on epidemiological evidence, it was felt to have a direct relevance to addressing broader stigma issues. This may be part of what Mr Messiah meant by the idea of pragmatic influence, but whether this is ‘not policy influence per se’ seems less obvious. Our conclusion points towards this being exactly the case (whether the depth of that influence is felt centrally or marginally). In a sense, then, and without wanting to overestimate such influence, the explanation for how this works may be provided by the Regional Director of UNAIDS, in pointing out that: “We have a notion that evidence must inform policy – but policy is often made on the basis of FBOs and media agendas – not evidence. The linear notion of evidence does not fly – popular notions of masculinity/sexuality and religion have the most influence on policy”.

b. Improving the access of key populations to HIV community services i. Changes in knowledge of HIV risk reduction achieved and contribution of strategic information activities The exploration of this question suggested that CHAA’s activities have greatly contributed to improved knowledge of HIV risk reduction, particularly amongst the most-at-risk populations served, but that the formal research studies per se have not played a great part in this. As conceded by one of CHAA’s research staff; “I’m not sure exactly how much the SI studies have informed the animator outreach work because of the timing of their coming out just recently”, as also acknowledged by UCSF. On the other hand, less formal intelligence gathered in mapping and outreach activities has informed the work, as explained by one

CHAA Programme Officer; “The way programmes are developed is informed by animators, but this is not really documented.” The result seems to be a highly appreciated outreach service, as well as a better appreciation on the part of other stakeholders, of what works and what is needed for these groups. Knowledge of HIV and strategies to reduce risks of infection is said to be high in the Caribbean, in general, but beneficiaries of CHAA’s work still feel the information, support is highly valuable to them. Some of these felt positive impacts are reflected in table 1, below. Table 1 Client survey findings: knowledge of condom use and its’ benefits, felt benefit of CHAA’s peer-outreach and the passing on of information by clients % (and #) Respondents claiming knowledge of condom use and its 98% (145) benefits

% (and #) Respondents who felt that CHAA activities have benefitted 97% (144) them directly in any way

% (and #) Respondents claiming to share what they learn from CHAA 91% (134) with peers Note: Numbers are self-reported ‘Yes’ answers and percentages are calculated as a proportion of the total sample of 148 respondents. Missing answers and ‘No’ answers make up the remainder of %ages to 100.

Whilst the fact that project animators interviewing their own clients about the project will likely contribute to some overly positive reporting, the degree of overwhelmingly positive feed-back, coupled with the UK evaluators’ observation of many interviews, leads us to conclude that the service is both perceived as very valuable by clients and indeed benefits them directly, as also reported by many key informants. For example, when asked if CHAA’s programmes are based on strategic information, Lyndale Weaver-Greenaway, of the Antigua Planned Parenthood Association, replied: ”Yes, I believe so, because they know the persons in the communities. I like the peer-model they use because persons don’t feel isolated and it is done by persons who are like themselves and who know their problems and concerns”.

Raising public awareness at the World AIDS Day Rally, Antigua, 2010

The client satisfaction survey, more in-depth one-to-one interviews and a few group discussions (such as in a ‘strip club’ in Barbados) all confirmed this general conclusion – CHAA’s outreach and education is indeed seen as of great value to improving knowledge and understanding of HIV risk and associated sexual health issues. A few illustrative explanations of how the services have helped clients gain better knowledge are described in these quotes by clients: “I learn about the usage of condoms and the sex risk. I also learnt how syphilis spreads.” (MSM, St Kitts) “I've gotten more aware of the STIs out there, and if it wasn't for the in house testing most girls wouldn't be tested” (Club-based sex worker, Barbados) “I read and research more about HIV and STIs” (MSM, Barbados) Against the background of this positive assessment a central challenge highlighted by Karen Blyth from Intrahealth was an apparently low HIV prevalence among sampled populations within some of the islands (for example Antigua). Whilst HIV prevention work coiuld be effective in such settings, work for people living with HIV was hard to develop. This is not withstanding that many HIV infections in the region likely remain undiagnosed, and frequent movement of people between islands, such that national prevalence statistics can be especially unreliable indicators of HIV prevention need.

ii. Beneficiaries’ behaviours

perceptions

changes

risk

Whilst there was really no questioning of the programme’s impact in terms of information and education around HIV risk, the question of the activities’ broader impact on behaviours was seen in more mixed terms. For example, Ian McKnight, of Caribbean Vulnerable Communities (CVC), reflected on this question in the following terms: “If you are asking about impacts at the community level and the animator programmes…? … my concern is more about how effective peer outreach approaches are in changing behaviours more generally. It’s a bigger question, as peer education (which is what it comes down to) tends to get whittled down to providing condoms, lubes and information about condom use. I wonder how equipped some of the animators are to deal with the deeper vulnerabilities and human rights issues and abuses which the groups face in their lives. CHAA’s voice really doesn’t get heard in terms of changing the laws. The problem is that CHAA really doesn’t have independence to act on this.” But, beyond advocacy for tackling the overall legal framework or deeper structural obstacles facing most-at-risk groups in the Caribbean, there were also a few respondents, who were not convinced that information, education and communications, even with the supply of condoms and lubricants, is sufficient to generate behaviour change. UCSF’s Principal Investigator pointed out that “Individually focused interventions don’t necessarily work in the long term – you need combination prevention”, but then added that the outreach may have had some impact “in combination with testing”. For example, Patrick Prescod, of Population Services International (PSI) in St Vincent explained; “I was looking at their IEC materials, which are simple and good, so that will help. People here have a high knowledge, but their behaviours are at a very low base. However, they funded SVPPA in the Northeast of the island for BCC work – I was part of it – and that aspect at least I can say it was effective. Of course, I may be a little biased…! [laughter]. The BCC aspect is very important and the IEC part is complementary”. When speaking directly to clients, however, the evaluation team felt that, on balance, the support, guidance and commodities were seen as very helpful for clients in reducing risks in their sexual lives. The great majority of clients asked in our satisfaction survey felt that they have indeed been able to change their behaviours and practices for the better, since contact with CHAA, as described in Table 2, below. Furthermore, a very large majority reported significant condom use over the last six months, as well as estimated that their condom use is now significantly higher than it was two years ago. The average self-estimated change would represent some 20 percentage-points’ increase in the proportion of their (penetrative) sexual encounters being with condoms, over the past two years. Adding a further positive reflection, Karen Blyth from Intrahealth also stressed that the CHAA working model, working closely with most at risk 14

populations as key programme operatives, was innovative and important for the region – and something not seen enough elsewhere. In as much as such work may be perceived as reductive (in terms of an emphasis on commodities distribution) the support and development was also appreciated as a significant contribution to behaviour change promotion practice. Table 2 Client survey findings: Perceptions of changes in own risk behaviours % (and #) Respondents claiming they have made changes in 76% (113) hygiene and sexual practices, since before contact with CHAA Average claimed condom use - as in estimated %age of (penetrative) sexual encounters in the last six months being with 89% condoms? Average estimated condom use two years ago - as in %age of (penetrative) sexual encounters two years ago being with 68% condoms?

A wide range of explanations or elaborations were given to illustrate these points, such as the following examples of quotes: “A lot – specially knowing my status and getting treated early to prolong my life span” (MSM PLHIV, St Vincent) “Every and anyway this is beneficial to my general health” (MSM, Barbados) “I always keep condoms with me at all times” (Club-based sex worker, Barbados) “I am now pregnant for my own decision, but I still use condom”. (Hispanic sex worker, Antigua) “I have never used condom in my country as constantly as now, and I had never done an HIV test.” (Hispanic sex worker, Antigua) “I practice myself to get comfortable in using condoms” (Sex worker, St Kitts) “It's me who puts the condom on to the customer” (Hispanic sex worker, Antigua) “Most of the time I use condoms with my main partner but other partners I will use the condoms every time” (Sex worker, St Kitts) 15

“Reduced partners and monitor my condom use” (Club-based sex worker, Barbados) “Using condoms always because I didn't use them before” (Hispanic sex worker, Antigua) “Using condoms even with my trust partner and take the HIV test every 6 months”. (Hispanic sex worker, Antigua) “Using condoms. I've made some mistakes from life and I'm not going back, I know better now” (Sex worker, St Kitts) We see no reason to doubt the veracity of the conclusion that beneficiaries’ perceptions of changes in their own risk behaviour have been positive, on the whole. We do not have access to any bio-data to corroborate these claims and impressions, or to judge whether these perceptions of greater safety translate into fewer infections, but we have no reason to doubt this either. Furthermore, whilst there are sceptics of the approach to peer education, it is important to stress that the animators provide a range of other services, including counselling, psycho-social support, referrals and practical advice, all of which is further explored below. Whilst the questionnaire did not probe more deeply into potential reasons for non-use of condoms or other protective behaviours, the high reporting of behaviour change was often accompanied with explanations suggesting earlier obstacles to safer sex. Whilst consistent condom use and partner reduction were the key strategies employed by respondents, common challenges suggested in answers included, for example, earlier misinformation on the safety of reliance on strategies like ‘douching’ or ‘faithful partners’ as non-risky. One respondent described: “I was under the assumption that my husband (who is positive) and I could have sex without protection, but this practice was changed due to the information I received from an outreach worker”. Other common obstacles and constraints included a lack of practice and familiarity with condom use, as described by two sex workers – “Before I didn't know how to put on correctly a condom and now I know. I had never seen lubricants before” and – “I am more confident about instructions and I protect myself more”. This was sometimes also linked to self esteem, as elaborated by another; “My self-esteem is higher, I feel myself I am valued although I am a sexual worker, and I have rights”.

iii. Beneficiaries’ perceptions of levels and quality of HIV service access As suggested in many of the quotes and reflections above, CHAA’s animator outreach programme is seen as well-adapted to the needs of most-at-risk populations, as proactive, confidential and personal, providing a range of types of support to clients, from advice and supplies for sexual health (HIV prevention in particular) to education, counselling and personal advise on difficult life 16

situations, but also referrals to official testing and counselling services, or to other support for housing or benefits – often accompanying clients and following up with support and advice. Francine Bess, at Barbados’ Ministry of Housing explained that: “…we might get particular requests and needs of particular individuals. In one case, a CHAA animator left a message regarding a person needing housing and asked us about protocol. So, I explained the procedure and she said she would send the person to come and see me.” The nature of the relationships will vary somewhat between the individual animators and the features of their specific target clients, but one Guyanese sex worker in Barbados described her view of CHAA’s outreach animator in a way which sums up the quality of many of these relationships quite well: “Speaking for female sex workers, a lot are there for different reasons and circumstances, but we have someone we can rely on, get psychic rest to sleep at night. We have a shoulder to lean on. The confidentiality is really great. They’re all we’ve got, so we really wouldn’t want to lose them.” Of course, the starting point is that groups most-at-risk of HIV infection were not generally receiving appropriate HIV and sexual health services in the Eastern Caribbean on any significant scale prior to the programme. CHAA’s own monitoring data (see table AT1 In Annex 4) shows that the programme now reaches around 2,000 clients per quarter, a great majority of whom also share information and learning with their other peers, as described in section b.1, above. (These are not insignificant numbers given the very small populations of the countries covered). Judging by our satisfaction survey of 148 outreach clients, it appears that beneficiaries perceive their levels of access to CHAA’s services as very good, particularly when compared to their perceived levels of access to other health services, where – depending slightly on the clients’ identity, life situation or type of health service in question – stigma and discrimination still pose major obstacles in many cases. Access to general health services were often said to be less of an issue, but one sex worker in St Kitts verbalised a fairly common problem noted about sexual health services, namely a view that that there is “too much gossiping”. As shown in table 3, below, clients rated CHAA’s outreach services as very good indeed, on the whole, and the great majority felt that CHAA’s animators are very accessible to them. Very common reasons given for this included “because the animator comes frequently” or “they are always willing to help and they are very kind” and – most frequently – “I can contact them by telephone when there is a need”. One client explained; “They try to put themselves in way which no other organization does to meet a particular need” and another even said; “Yes, it's like I have you at my beck or call!” In fact, the accessibility of CHAA’s animators appears so great that our main concern would be for the psycho-social support and well-being of the animators themselves and the degree to which they can be supported to balance privacy and personal space with work. 17

Table 3 Client survey findings: Perceptions of levels and quality of service access Average rating of CHAA’s outreach on a scale of 10 (0=bad; 10=Excellent)

9.3

% (and #) Respondents who say they feel that CHAA's animators 88% (130) are easily accessible

Average rating of how clients feel they access other health 5.8 services (0=difficult; 10=very easy) Note: Other health services included general health services, as well as other STI or HIV services

As mentioned earlier, an important dimension of the animator programme is that of facilitating access to HIV testing and counselling and to other health services, such as for sexual and reproductive health or family planning. CHAA is indeed seen by clients as helping to improve their access to services, more generally, and table 4, below, sums up the proportions of clients surveyed who have been referred by CHAA for HIV testing and counselling, to other health services and what their rated experiences of these referrals were. Table 4. Client survey findings: Perceptions of referrals facilitating access to testing, counselling and other sexual and reproductive health services % (and #) Respondents who say they have been referred by 76% (113) CHAA animators to HIV testing and counselling

Average rating of the experience of the testing and counselling (0=bad; 10=Excellent)

8.9

% (and #) Respondents who say they have been referred by 67% (99) CHAA animators to other services

Average rating of the experience of the referral and services 8.9 received (0=bad; 10=Excellent)

In terms of referrals for HIV testing and counselling, a large majority of clients reported that they have been referred by CHAA staff and it seems clear that the availability of rapid testing is appreciated, or – where not available – wanted, as one respondent in St Kitts explained that “the services were OK, but the time for 18

the results took too long. Immediate response would be better”. In Antigua, where rapid testing is more available, another respondent rated the referral experience as excellent “…because they give the test results at the same time”. There were occasional problems, however, as a sex worker in Barbados explained that “…they had a problem with labelling blood samples I had to repeat test due to the mix up”. For referrals to other health services, such as for sexual and reproductive health or family planning, many clients felt these were very important and valuable indeed to them. Furthermore CHAA’s engagement with other service providers, such as with Planned Parenthood Associations as implementing partners, seems to enhance their service provision to, for example, sex workers or gay and bisexual men. One client of the MSM animator in Antigua explained that “the staff at the Planned Parenthood are very friendly and give information when needed”, whist one Hispanic sex worker said she rated the experience highly “because I took a free smear test and in another place I had to pay 100 Dominican Pesos”. In terms of referrals to support for other issues, like social needs, it is often less straight forward and one PLHIV from St Vincent explained that “after being referred it takes a lot of time if it's social support to be accessed.” A key and appreciated strength of CHAA’s animators’ referrals was said to be the close hand-holding and support in making sure the service is accessed, as illustrated by one sex worker in St Kitts; “…my animator always accompanies me and the persons I met were really nice to me too”.

iv. Results and effectiveness of community-based rapid HIV testing in Antigua and Barbados Community-based rapid HIV testing has been well received among a range of stakeholders in both Antigua and Barbados. The prevailing impression is that the roll-out of this work has a had a potentially significant influence within MoH testing facilities albeit that in part, and especially in Antigua, structural and organizational issues have curtailed the reach and effectiveness of rapid testing activities. These complications were observed by a number of interviewees in Antigua. Lyndale Weaver Greenaway of the Antigua Planned Parenthood Association noted: “It was good because a lot of people got tested. Quite a few people were trained… The NAPS also still do testing in collaboration with Scotia Bank, but we have to wait for NAPS to get accredited and then we can get accredited. There was some problem with the head of the pathology lab and he had some reservations. But, what is good is that when NAPS go out to do their testing we – the other partners – can come along and practice”.

Ms Greenaway was especially pleased to have undertaken VCT training and was also impressed that key representative from the main organizations working on HIV, sexuality and sexual health in Antigua had also been trained. This helped to build the perceived legitimacy and potential impact if rapid testing. The main benefit is that other testing regimes require clients to wait for some time for results, which both causes and anxiety and can be a reason to decide against testing in the first place. One aspect of CHAA’s approach to rapid testing was to emphasise the use of lay persons for conducting tests and counselling. The Director of the Antigua NAPS, Ms Delcora Williams, explained that: “… before CHAA came our testing would be kept in a book, maybe 30 per quarter. Now we have had to computerise everything and there is so much testing being done. We are going through a lot of stress in these three months as CHAA is downsizing. Now we test in the 100s per quarter”. The implementation of rapid testing was somewhat undermined, however, by delays in accreditation at the national pathology lab, where existing HIV testing takes place. Issues around control of testing activities within a decentralizing programmatic framework (which places testing in the hands of community workers, animators and external agencies) appear to be a significant part of the problem in Antigua. Decentralization is important to the community-oriented ethics of practice within the CHAA rapid testing model; yet, it may similarly constitute an obstacle to accreditation from the lab (which de-facto appears as more centralized and ‘medicalized’ in orientation). Ongoing advocacy is required from both CHAA and within the NAPS in order to extend consensus for rapid testing in Antigua, including the national pathology lab. The PO in Antigua reported that some progress is being made in this regard by the current leader of the NAPS, but it is nonetheless important that CHAA continues to play a role in developing the programmatic environment for rapid testing. Where rapid testing has taken place in Antigua, such as on the special days mentioned above, ‘simulating’ a more comprehensive roll-out, there has been high turn-out. Indeed Ms Huggings from Gender Affairs noted that the numbers who come for testing on these days makes implementation difficult, “making it hard to follow counselling protocols when so many people are keen to be seen”. She further noted that in a small island there are concerns over confidentiality in counselling, such that “even where confidentiality protocols are observed there are simply issues with people knowing one another, or knowing family members and so forth”. It is not so much that confidentiality is not observed by counsellors, but that clients may struggle to understand ways in which this is respected, especially given that, for many, counselling with HIV testing may be a new concept. The notion of confidentiality in counselling may also sit uneasily with the ‘public’ nature of an RTC rollout event.

The development of community-based rapid testing in Anitgua was especially enhanced by having a consultant who was employed through Intrahealth living and working on the island, Vincent Guilin. His work was well regarded by many respondents, and his level of technical expertise and work ethic was seen to have added to the development of expertise among others. This also presented some problems as regards the expert’s situation within the overall CHAA framework, which was felt by Intrahealth to be ambiguous and sometimes ambivalent, which may in part help to explain why the work carried out on rapid testing in Antgua was not well replicated in other settings. Community-based rapid testing has not been rolled out and decentralised in Barbados, although rapid tests are provided on specific days and major testing events, as is the case in all of the four countries. Some people, including CHAA staff and animators, have been trained in rapid testing (as have colleagues from the other islands) and much is in place for rolling it out. Mr Basil Hunte in Barbados (responsible for HIV testing and counselling in the MoH) noted: “The relationship with CHAA has been based on HIV testing in specific areas. We have 8 polyclinics. I have a lot of respect for CHAA’s animators who do the ground-work. The 8 counsellors in the clinics are not willing to do it… CHAA has a unique relationship to the nightclubs and the groups. My main concern is that the painstaking groundwork that has been invested in this doesn’t get wasted”. So, the approach has been piloted, but not yet rolled out as a community based model. In fact, it would be a mistake to describe it as ‘one model’ and, as pointed out by CHAA’s Executive Director, such a model “is not on the table yet, as physical and tangible. There is nothing concrete to share with the region. There is lots of excellent work in-country on counselling and testing etc., but demonstration of this beyond countries falls down”. Progress is definitely being made and many of the constraints are external to CHAA’s direct control. Beneficiaries on the ground often speak very positively about their experiences of rapid testing and CHAA’s animators are seen as greatly facilitating access to testing generally (whether rapid testing, more standard VCT or STI testing). Yet, it remains difficult to assess the full results and effectiveness of the specific contribution of the Alliance, in the absence of a more structured and formal rollout of community-based RTC. CHAA monitoring data show relatively high and growing numbers of referrals, generally, but this is not disaggregated between tests that are rapid or VCT.

v. Role and value added of the partnership with Intrahealth The partnership with Intrahealth and their locally posted expert (generally referred to as ‘Vincent’), has clearly been seen as a great contribution, as Intrahealth's expert reportedly played a key role in pushing the agenda forward in Antigua against many odds and challenges. Whilst this was visible and

appreciated far beyond CHAA, the benefits were perhaps most keenly felt by members of the staff, one of whom related the experience of how Intrahealth; “… started the whole process. A first technical report was produced and then a model was defined and people were trained. As far as I can see, it has been very good. The [Antigua National AIDS] Secretariat has told us that they have increase testing levels enormously and they have gone out with mobile units… going to mobile testing units is fine, but it may have some issues with stigma, since it is quite visible and public. I don’t know how it has gone in Barbados. All the people interviewed described Intrahealth as coming out to them and explaining the programme, so that partnership was appreciated by partners”. Intrahealth were engaged because of their experience in this area. Reportedly, the idea was that the regionally based adviser would be replaced by a local person, but this did not happen. CHAA’s Chief of Party, Jane Armstrong, felt that, on balance, “the expertise was needed. And rapid testing has expanded and gone forward a lot”. Vincent Guilin’s personal expertise was widely and highly appreciated and many felt that this reflected well on the organisation and the partnership. Delcora Williams, the Director of Antigua’s National AIDS Programme Secretariat, described it in the following terms: “It was excellent. They helped us a lot with trainings, including in rapid testing with CDC and I have trained others here since then. “If we were not working with CHAA I don’t see the programme working at all…They are already on the ground and have excellent contacts…money just can’t buy that. They have already developed trust with those populations… they are very professional” The value of the support from Intrahealth went beyond the medical and ‘hard’ technical aspects of the work, but also the psycho-social dimensions of the work, as described by Ms Huggings, of Gender Affairs in Antigua: “Intrahealth were very helpful in promoting understanding of how stigma and discrimination are inherent within testing issues, so it was more than a model of just testing per se”. The relationship also helped CHAA with linking to CDC as partners and staff, animators and lay persons were sent from the Caribbean to North Carolina (and also to Barbados) to train in fingertip rapid testing, along with counselling training for work with MARPs. Against the background of these positive assessments, Karen Blyth from Intrahealth nonetheless felt that there were some less positive issues in the working relationship with CHAA. Some initial challenges in the managerial rapport were described, but the relationship was also said to have improved as the project progressed. Karen Blyth emphasised how much she admired and supported CHAA’s work with most at risk populations, and especially more widely 22

the Alliance’s approach to developing and supporting work with local partners incountry. Indeed, she would have liked to have seen a stronger and more defined role for the Alliance secretariat within the work carried out regionally. Overall, the partnership has had a very positive impact, particularly in Antigua but also on the other countries, if less directly so, through this training and promotion. The relative success of this may be put down to good leadership and communication skills combined with strong technical expertise, drive and commitment of the key expert deployed, as well as good institutional networks.

vi. Effectiveness of bi-directional referral systems in Antigua and Barbados Referral systems in Antigua and Barbados were broadly reported to be effective and evolving. As the programme officer in Barbados, Teddy Leon, noted: The programme was very ambitious. There was no bi-directional component in the rapid testing roll-out, but there was in the BCC to counselling and testing. It is working in practice, but it is a bit more organic than the word sounds. It is still part of the components, but the relationships for the referrals are built on the BCC and they can go in different directions, to testing, to housing support and so on. Mr Leon noted that bi-directional referral is not always a discrete or even directly tangible aspect of programme activities per se, but something that often happens in the course of the organic social relationships that animators develop with the people whom they work with – in the course of BCC activities. This mode of referral was widely reported in the interviews conducted by animators in each of the four islands in which CHAA operates. Most often reported referrals were to HIV testing services, as noted by a respondent from Barbados: “Yes, the animator referred me to testing at Hero Square on regional testing day”. Aside from occasional testing days, the more common referrals are for regular VCT or STI tests. The manager of the counselling and testing facilities of the Barbados MoH, Mr Hunte, noted: “In terms of the referrals, CHAA animators can refer directly to the polyclinic or directly to counsellors, who know the issues. However, a lot of members of these groups are not comfortable walking into the clinics on their own and sometimes they are accompanied by animators”. The work of animators is a key component of effective referral, offering personal support in accessing HIV prevention services, for instance, which is especially important in building trust and confidence in the relevance of testing referral for risk populations. One Guyanese migrant sex worker in Barbados explained that

her “… testing experience was great, even though the wait was a bit ‘painful’, but the counselling and support was really great.” More broadly, our client satisfaction survey suggested a very positive view of CHAA’s role in referrals. When asked “Have you been referred by CHAA people to HIV testing and counselling?” Seventy-six percent of respondents confirmed that they have been referred by the animator. Furthermore the average rating of this referral experience (on a scale of 0-to-10, from poor to excellent) was at 8.9 – in other words and on the whole ‘very good indeed’. There were, of course, some less positive experiences, particularly related to the place of testing not being conducive, or the wait for results (although the latter was not examples of rapid testing referrals). On the whole, the negative experiences reported were few and one respondent rating the experience with a ‘9’, explained; “I have never been tested before, because I was afraid, but after I was counselled, I made the decision to go for the test to know my status”. Similarly, members of implementing partner organizations reported that CHAA had helped to generate a referral environment that facilitated their referrals of most at risk person to MoH testing services. Rev Karen Brotherson from HHH in Antigua noted: “Because of the referral system established with CHAA, relationships of trust and confidentiality have been developed – and privacy has been maintained in referral. I can now make referrals on behalf of MARPs who do not necessarily want to be identified by name – I arrange for confidential referrals by calling the testing centre and arranging the appointment”. This positive view of animator and IP driven referrals for HIV testing with MoH’s and NAPS ought to be qualified, however. In some contexts prejudicial views towards animators have not been entirely overcome in some NAPS. One informant noted, for example, that a Spanish speaking animator who conducted testing work within the NAPS with clients she had referred there was asked to conduct her work in English. It is possible that this reflects an underlying prejudice toward Hispanics. Whether or not that is so, this hampered some aspects of referral by animators to the NAPS in Antigua and also suggests that referral to animators from the NAPS may be somewhat lacking. The CHAA PO in Antigua, Louise Tillotson, noted that even though CHAA has strengthened referrals, often animators come across problems for which they cannot really refer – because of lack of specifically relevant services to refer to on small islands. As such referral is not always a viable working model. Yet, due to close relationships developing over the programme (though joint trainings etc.) referrals are sometimes made for various reasons beyond sexual health, as described by Ms Francine Bess, from the Ministry of Housing in Barbados: “PLHIV get referred to VCT and the Ministry of Health refer cases to the Ministry of Housing. Sometimes the ground-base [CHAA animators] will 24

contact me directly and then we reverse it and go to the Ministry of Health. For example, we might get particular requests and needs of particular individuals. In one case, a CHAA animator left a message regarding a person needing housing and asked us about protocol. So, I explained the procedure and she said she would send the person to come and see me”. It is worth noting that, although the Alliance was closely involved in pushing the rollout of rapid testing in Antigua and Barbados, rapid testing is also being implemented extensively in other countries, including St Vincent. Similar referral practices exist in all programme countries between peer animators and testing services, although bi-directionality – where it exists – is more organic than characterised by structured formal referrals. In St Vincent’s main government testing clinic, where one of the programme’s outreach workers is targeting clients in the waiting room, Dr Margarita Tash noted that they “…haven’t had to refer anybody to their [CHAA’s] office, but if a person needs peer support we refer them to the Alliance peer worker here”. Additionally, some pointed to a need to have regional referrals, because of a very high level of migration and movement of people between islands. This could be formalized more, for example with booklets about services in different islands, services in other countries and strengthening contacts between service providers in different countries. Overall, it would be fair to say that the concept of bi-directional referral systems is not evenly understood, nor indeed systematised, yet referrals are happening extensively in multiple directions and improving as actors in the response get more familiar with services and engage with each other through the programme. The community animators play a central role in this, but it remains unclear as to whether a more formalised system would improve the work in all settings. It seems more advisable that different countries may want to develop locally defined protocols and pilots, then assessing these and sharing practices across countries at regional level.

c. Capacity building and civil society involvement i. Strengths, challenges and responsiveness of CBO capacity building approaches There was a major emphasis on training workshops, from respondents in discussing CHAA’s capacity building approaches, but some also pointed to – and valued – the on-going relationship and accessibility of support and guidance from CHAA. The organisation puts a strong emphasis on organisational capacity aspects, as well as trainings in technically substantive issues, such as on counselling, stigma and discrimination work, rapid testing etc. Bishop John Collins, an interviewee in St Kitts, was especially appreciative of this aspect of CHAA’s work and spoke about how the training with FBOs in St Kitts had helped 25

members to realise their own prejudices towards people living with HIV. A person living with HIV came to St Kitts from Jamaica, supported by CHAA, and she had offered training and the face-to-face with church organizations, helping to or catalyse, a profound and culturally transformative process of change for the FBO participants. There is still much work to do in addressing stigma and discrimination but it was noticeable in St Kitts that FBOs are playing a significant and supportive role in work with PLHA (whilst other work with other MARPs such as MSM and CSW seems far more sensitive in these contexts). This evolving change in attitude has been directly supported by CHAA’s capacity building work, with activities that help to change attitudes and so forth. CHAA’s Technical Director explained how, when setting up with implementing partners, CHAA first put out an open call, asking organizations to put forward service relevant proposals. The organizational development specialist from CHAA visited the different countries to interview organizations and assess their capacity, before grants were awarded. In addition, all implementing partners must do CHAA’s M&E training and then strategic planning, proposal writing and so forth. This investment was appreciated by CBO partners and beneficiaries, but there were some mixed feelings about some aspects of these procedures among respondents, nonetheless. Reverend Karen Brotherson, from HHH in Antigua noted: “Writing a proposal is easy, but the M&E and implementing is difficult – takes a lot of work. CHAA has taught the importance of M&E. At the end of the day everything is tied to M&E – it becomes something holistic in day-to-day activities – it is very ‘impactful’. Prior to CHAA all the IPs had trouble in understanding M&E. With a better understanding funding proposals become more viable”. For Reverend Brotherson CHAA’s work had helped to change the style and to some extent the ‘culture’ of report and proposal writing among IPs – potentially helping with sustainability. And yet she also noted that without CHAA’s financial support many of the CBOs in Antigua were unlikely to be sustainable, since other sources of funding are limited. Whilst CHAA’s work has certainly helped to expand the capacity of CBOs in terms of developing strategies for sustainability the broader funding environment is not necessarily replete with opportunities for sustained funding of work. This is especially so because a number of the CBOs that CHAA work with are quite small in scale, for example in terms of the numbers of people who are involved in them. As such, they may struggle to find appropriate funding environments as absorptive and managerial capacity for funds is organizationally weak. This is reportedly the case even among the larger of the CBOs with whom CHAA has worked. The CBO UGLAAB (United Gays and Lesbians Against AIDS, Barbados) has been among the earliest beneficiaries of CHAA support in the region, both financially and in terms of a range of training and capacity building activities. This was appreciated by the people who run UGLAAB, who were 26

especially pleased to find people from the own communities in the role of trainers and counsellors (in VCT work). And yet there is mixed feeling among people involved in UGLAAB vis-à-vis the sustainability of CHAA’s input in terms or organizational strengthening and capacity building. Some interviewees would have liked more sustained input in terms of infrastructure costs. UGLAAB is currently based in a small window-less room in the NAP, and whilst the provision of such premises to some extent indicate a positive inclusion of UGLAAB within NAP activities members nonetheless felt somewhat marginalized in this environment (not being invited to meetings with donors, or even being allowed to use staff toilets, for example). This whilst UGLAAB is in many ways in intrinsic to the response to MSM within the NAP the structural position of UGLAANB seems to undermine this commitment and it is questionable whether UGLAAB would be able to continue the development of this work without the ongoing support of CHAA. This was similar to the case with the SISTA programme in St Kitts. In this instance civil society was developed by working closely with women working in factories, taking them through a series of trainings, adapted from the US-based SISTA model for women’s empowerment (see page 28). This work did not support and existing CBO per se, but rather aimed to develop capacity among a group of women, who might then work collectively for women’s health and welfare thereafter, within the specific environment of their employment. The programme was very well received among respondent in St Kitts, both from the NAP and the MoH, and yet the question of what happens to the women now that they have successfully graduated from the SISTA programme was also a concern among respondents. Education and training is important but sustained self-efficacy and organizational development take a long time to institute, and for some there was a concern that CHAA must continue to play an ongoing work in developing CBO capacity. The culture of inclusion of MARPs with national AIDS programme planning and MoHs throughout the countries where CHAA is operating is nascent and uneven. Whilst CBOs are playing an increasingly stronger part in developing programmatic agendas for HIV prevention the overarching structural support of an organization such as CHAA (with the EC-CAP remit) is still a necessity within the region, not only in order to continue both the development of civil society organizations’ capacity but also to advocate for broader recognition and support for civil society groups within national programming and planning.

ii. Contribution of the grants programmes to results The programme’s results in increasing access to HIV and AIDS community services for most-at-risk populations through evidence-based programming are supported by CHAA’s grants programme in several ways. Complementary types of implementing partners – such as community care groups, family planning 27

associations, faith-based organisations or fledgling sexual minority support groups – can address different aspects of response (such as PLHIV support, focused HIV prevention outreach, improving linkages between HIV and SRH services, testing counselling and referrals as well as addressing stigma and discrimination). Grantees on most islands include a range of different types of implementing partners, who address these different – and sometimes multiple – aspects, whilst engaging with the national programmes and in referrals between organisations (which, as we have seen, are strengthened by familiarity and relationships developing between organisations in connection with training workshops for example). In addition, the engagement with CHAA animators and Programme Officers can also strengthen links and the ‘overall coherence of the response’ in locally appropriate ways that ‘build on what is there’. CHAA’s onward granting is significant across this range of activities, and plays a role in supporting CBOs in the region that would be difficult to carry forward otherwise. As a PANCAP and CHAA board member noted: “I’ve always said that there are certain things that CBOs can do better than states and things that states can do better than CBOs. Reaching the vulnerable groups and into the communities is something CBOs are best placed for. That’s why PANCAP looks to CHAA in this. It has a real niche”. Onward granting has enabled a responsive adaptability for CHAA, in terms of reach to small organizations that work closely with vulnerable populations. State health programmes and regional structures such as PANCAP are not well placed to undertake such a remit and as such CHAA’s funding partnerships are playing a unique and vital role in terms of developing the HIV response at community levels. A key question, however, pertains to the sustainability of this mode of granting. Ian McKnight, Caribbean Vulnerable Communities, noted of CHAA’s onward granting: “Yes, it does contribute. It has enabled really important work to take place which was only conceived of before. The issue is more ‘what’s the plan for after - and during?’ The requirements on the groups have changed radically over time, some of it to do with personnel changes at CHAA, reporting requirements, data collection, financial control requirements etc”. These concerns parallel those raised in respect of CBO capacity building, in which case issues arose regarding the ongoing support of CBOs, beyond a two or three year relationship. Similarly with onward granting, concerns were raised about the time-span on granting activities, particularly in environments where other sources of funding are scarce and where the state is unlikely to be able to sustain activities of this kind. As noted by a national AIDS programme co-ordinator, the NAS doesn’t have the resources for ongoing work with MARPs and that CHAA is vital for this, because it brings resources and is premised on a particular model of supporting NGOs. In 28

this sense CHAA performed as almost an extension to the national programme, with onward grating a vital component of planning at the state level. In these circumstances a considerable weight is being attached to CHAA’s onward granting with the EC-CAP countries, as an aspect of the different national responses. This is especially complicated because it tends to orient work with MARPs or key populations toward a somewhat ambiguous relationship with national responses. As noted MARP involvement in national AIDS programmes is uneven throughout the EC-CAP countries, and even where UGLAAB has been brought into the NAP working environment in Barbados, for example, it is still excluded and poorly funded at many levels. Other organizations in other countries occupy even more indeterminate positions. Whilst the expansion of the role of IPs with whom CHAA works in national HIV prevention has certainly been significantly augmented by onward granting, there is a also a concern that this granting has become assumed, and that work with MARPs in particular has not been sufficiently integrated in national AIDS programmes, other than via an expectation that CHAA will continue to support this work. In one way or another this scenario was common throughout the four countries where CHAA works – and is indicative of both the success of CHAA’s work in supporting work with MARPs whilst also highlighting ongoing problems faced with institutionalising HIV prevention, health an rights work with most at risk populations within regional state infrastructures. This problem is especially complicated because of the weakness of civil society in the region. As noted, a number of respondents from CBOs felt that the requirements in applying for CHAA onward grants were somewhat onerous, beyond their organizational capacity. A representative from one of CHAAs IPs noted on onward granting: “The procedure for accessing funds is tedious but effective. The only problem is that maybe it needs to be adapted for the Caribbean. Because most of the donors are in the US the model is very US-based. The CBOs have very little knowledge of donor agency requirements – and the requirements are too complex and people give up. CHAA training on these issues has helped to some extent. With AmFar it was a little easier because whereas with CHAA there is an expectation to write a proposal, but with AmFar it is more like filling in a form – more structured”. These concerns were similarly reflected by CHAA’s Chief of Party, Jane Armstrong: “Working with the IPs (CBOs) has been difficult. Initially the EC-CAP model was to work with 32 groups. This vision, looking back, was too grand in scale. A better way to do things might be to run a small grants scheme – funding CBOs for smaller scale specific projects that they can manage (and more easily manage the finances for e.g. such as making a

video). This gives people experience of managing monies, but on a scale that they can easily account for – without bureaucratic complexities”. The realities of granting to CBOs as IPs in the region have been an important lesson learned though the implementation of EC-CAP and the above point about working through smaller grant schemes - building up organizational capacity in a more measured manner through onward granting - is especially salient. Most sub-grantees to-date would not be in a position to continue their role within the HIV response without CHAA’s direct financial support (and the organizational support that goes along with this). Diversification and expansion in the total number of sub-grantees has been difficult because granting procedures have been too complex and the sums of monies involves too large at this stage for many of the potential IPs in the region. The next phase of EC-CAP ought to take these issues fully into account, taking forward a longer-term vision for the development, support and growth of CBOs through onward granting, whilst also implementing a new strategy for more explicitly (but subtly) managing common state expectations that work with MARPs will be supported outside of state resources. The plan of onward grating in the next phase of EC-CAP is especially important in these regards, and finding the right balance between appropriate levels of granting support and reduction in financial dependency among both CBOs and state organization ought to be a specific strategic component of planning.

iii. Relevance of the approach in the Caribbean context Questions regarding the relevance of the CHAA approach to the Caribbean context and region elicited diverse and contesting responses among different informants. Overall, there appears to be a marked differentiation vis-à-vis CHAA’s perceived strengths and credibility within regional HIV prevention policy and activist fora, on the one hand, as compared to the view from national contexts, on the other. In-country, there is clear influence within national AIDS programmes and CHAA activities were often spoken of very highly. Gardenia Richardson, the National AIDS Programme coordinator in St Kitts, for example, stressed that she could not speak highly enough of CHAA and that, indeed, there was almost a feeling that CHAA was a part of the National AIDS Programme, illustrating this point by saying it was hard for her to imagine the programme without CHAA. It was noted that many NAPs (especially in Antigua, St Kitts and St Vincent) typically say that they couldn’t have done anything with MARPs without CHAA. This view was similarly echoed in-country by a number of respondents, such as: “Yes, it is working in the region. What CHAA is doing that I like is that they go out and meet their clients where they are. They are filling the gap that the clinics are missing”. Regional relevance, in these terms, arises out of filling a gap in the region – with CHAA providing targeted services for MARPs that are otherwise not as 30

developed within the regional HIV response (outside of some key locations, such as in Jamaica or the Dominican Republic). Another member of CHAA staff reflected on the difference that CHAA had made in this respect: “It has made a big contribution, especially when it comes to MARPs. The first studies were done by CAREC and PAHO in the 90s on MSM and sex workers, but nothing as consistent and with outreach as until the Alliance programme and USAID started this. The regional element works because of networking and pooling skills across all the small islands. It wouldn’t be cost-effective to do this separately in each place”. Regional linking and sensitivity was also appreciated by Azilla Clarke from the MoSS in St Kitts. She had been particularly involved in the development and adaptation of the SISTA model of working with factory-based low income women in St Kitts (the most prominent example of HIV prevention model adaptation in the CHAA regional programme). Ms Clarke was especially impressed by the regional sensitization that was intrinsic to adapting the SISTA model to work in St Kitts – the working model was not simply imposed from outside, but rather thoroughly adapted to the context, involving the detailed development of workshop materials that included examples taken from real life experience in St Kitts and the region. This positive view of the care and attention taken in adapting the SISTA model to St Kitts and the region was common, but not universal among evaluation informants. Sybil Allen-Jones from the International School of Nursing in St Kitts (who provided psycho-social support to CHAA animators) felt that not all of the people involved in CHAA’s work were regionally sensitive and that, specifically, some people involved in the adaptation of the SISTA model did not necessarily understand the region well. Indeed, in respect of regional relevance she felt it important to stress that each island is different, so that the regional relevance of CHAA was seen as resting in programmatic sensitivity to the heterogeneity of the region. This was – for Ms Allen-Jones – lacking, especially because the overall staff profile does not necessarily reflect regional diversity, being somewhat biased toward staff from Trinidad, for example, with less recruitment ‘in country’. The Principal Investigator for UCSF, Janet Myers, felt that the model was very challenging for the region and reflected on a sense that countries “pulled in different directions”, whilst she also added, “but now we can work in those countries to develop strategic approaches together? Yes: Relationships had to be built and there are few people to work with.” Differentiation between positive national perspectives on CHAA, as compared to variable regional reception and legitimacy remain key issues for future development and for building a wider influence on the basis of strong field-level work. It is important to stress that how CHAA is being viewed is shifting and improving, and its work is increasingly appreciated at the level of, for example, the PANCAP.

iv. Efficiency and achievements

sustainability

activities

and

One of the most salient symbols regarding the potential sustainability or otherwise of CHAA’s achievement and activities in-country and in the region was the closing of CHAA country offices during the course of the research. Questions concerning sustainability were explored against the background of an apparent ‘winding-down’ of CHAA’s activities, which was profoundly unsettling for many of the programme staff and animators, whilst also signalling a confusing message to many state and non-stake stakeholders, who were unclear whether CHAA was closing for good, or whether ‘apparent closure’ was an consequence of delays within the programmatic funding cycle, which would ultimately be resolved. An important starting point about sustainability, in a concentrated epidemic setting, relates to the focus and quality of the prevention work and the programme’s ability to ‘follow the virus’ in the right places and at sufficient scale. As a peer stakeholder in Barbados, Francine Bess of the Ministry of Housing, pointed out: “… it can be sustainable, providing CHAA can get sustained funding. If we can prevent transmissions it will also improve sustainability overall”. To reach the enough of the populations most at risk, it is crucial to be able to combine skills and capacity to reach and engage these groups (which CHAA clearly has) with intelligence about their locations, networks and extent (or population sizes), including a grounded understanding of ‘population boundaries’ overlaps and dynamics. The task has been started in earnest and a good potential is there. However, it was noted of the Infections Disease Clinic in St Vincent, “…they need some more time. For example, with MARPs we are supposed to do size estimations and it would be good to partner with the Alliance. I don’t know if they have something in place and some of the animators are a bit concerned. I’m not hearing much about that…” CHAA PO in Barbados, Teddy Leon explained that “governments will never do this here and get their hands dirty with ‘those people. There is no other way to do this if you want to reach the communities.” Furthermore, Carl Browne, Director of PANCAP and member of CHAA’s Board, felt that, “very frankly, the only way is to work very closely with national programmes and to be seen as “part of” national AIDS programmes and then to become seen as a country responsibility”. This level of sustainability and integration is certainly some way off still, for the programme as a whole, but there are also very promising sings, which make this orientation a viable part of a longer-term sustainability strategy. For example, it was suggested by a member of a National AIDS Commission that “… the model is really valuable because of their expertise with MARPs. We just don’t have it. CHAA has a considerable comparative advantage and they are not afraid to share their skills and so on.” In discussing CHAA’s role in the broader role national response, she also explained that the NAC “… are considering having CHAA lead a civil society grants programme, as we only have 4 technical officers here. We have got recent approval from the World Bank and approval from the Commission”. 32

v. Effectiveness and relevance of management for objectives The International Alliance and CHAA took on a big job in bidding for this programme and the work plan was both ambitious and complex in terms of partnerships, recruitment demands, staffing structure and operational procedures to carry out the job across four small countries overseen by a regional office with support of the International HIV/AIDS Alliance from the UK. The effectiveness and relevance of how CHAA and IHAA have managed the programme can be addressed from both external perspectives and more internal reflections, which may illuminate some strengths and weaknesses in sharper contours. The evaluation team are clear that we are not passing judgement in terms of a broader organisational review and we recognise that EC-CAP is but one (if the main) programme implemented by CHAA. Hence, we try to focus these reflections and comments on how management relates to the achievement of the three key objectives of EC-CAP itself. The main findings on contributions, relative successes and challenges of SI activities are discussed above, but some further analysis of the management of this area of work may be helpful. A major challenge already noted in the area of SI (research and M&E primarily) has been considerable flux in structures and responsibilities, as well as delays in recruitment and staff turnover. Even at the time of review the new CHAA Research Director was not in charge of M&E, with this function being managed on a consultant basis from Canada. Whilst this may be a temporary and transitional arrangement, we were also told that recruitment and set up took a long time to begin with and we were left with a feeling that the research and M&E components are not closely integrated with each other, nor closely enough with CHAA’s operational work. According to a member of the research team at CHAA, the SI studies go through numerous drafts and changes through various ‘tortuous reviews’. First the Evaluation Director edits the report, then; all the POs of the EC-CAP Team review it, the CHAA Senior Management Team then reviews it, the Policy and Programme Development Manager, then it goes to IHAA and finally USAID. In one case, IHAA reportedly turned around and objected to a study on the basis of the design of the study, although they were said to be co-authors, technically codesigners and should have inputted at the design stage. It was then said to have to go to an IHAA Research Committee, which apparently left CHAA hanging for very long and then had to contract a consultant to comment, in the end. Most of the comments were said to be fine in themselves and except for the fact that they can’t actually be acted on at this stage. So, all this was felt to be quite frustrating and summed up in the conclusion that “it is not the science driving this, but external relations and internal issues slows it down.” In a different conversation, a senior member of CHAA’s management explained that IHAA’s technical support was ‘pulled out’ a few years back and, whether or

not this was due to financial reasons, this appears slightly unfortunate, as this support and resources have been highly valued in the past. In terms of the outreach programme and improving access to services for MARPS, external perceptions are broadly that CHAA is going a very good job, being very professional and running a fairly lean and ‘tight ship’. For example, the Assistant Director of the NAC in Barbados explained; “When Teddy [CHAA PO] first came, CHAA was willing to modify their own M&E forms – now, that’s a hallmark of a true partner! They have been able to identify the MARPs which gives us an idea of the programme coverage.” So, if you ask ‘has CHAA fulfilled its mandate?’ I would say ‘Yes, unequivocally!’ ” Another respondent from PSI (a peer organisation) in St Vincent said; “From my perspective it looks good. It is never good to have a too big structure and theirs is simple and effective”, whereas a respondent from of CVC, a regional peer network felt that “If anything, I see some things like ‘who can sign off on things?’ or ‘what the animators can do etc. without checking first’. I question whether, for example, if an animator was interviewed impromptu by a journalist, they would feel empowered enough to seize that advocacy moment?” CHAA in-country Programme Officers had nuanced views about the strength of the management, which will be elaborated a little more below, but one PO felt; “Yes it [management] has been [effective], although maybe I am privileged in that I have a lot of support here at hand … compared to other countries.” In terms of managing its’ support to civil society partners and capacity building of implementing partners, again external views of CHAA were broadly positive, with examples of views including: “[We were] exposed to different levels of CHAA’s structure, sometimes with some confusion. In country, having a PO has been very suitable, and has worked well”. “There are different levels, but it is about team work and I can talk to anyone, just about”. “It [management of support] has been efficiently done. You can call anyone in Trinidad … and the local office here is very supportive”. “Management has been adequate and effective – efficient. There has been lots of scrutiny in management – which has been a part of building capacity”. “My project has had some problems. We were supposed to start in December, but signed in February. Then, we only got the funding in May… But, locally, we have… [the CHAA PO] and the PA so that structure was good. They had the right persons.” According to several CHAA in-country POs, management has, however, been somewhat of a challenge in terms of ‘structure’ and insufficient devolution of 34

decision making. One of the PO staff also pointed out that the involvement of animators with the organizations structure in a more inclusive or formal way has not happened yet. Whilst management messages were felt to not always come in clear lines, it was also reflected that POs, over time, have become like country managers. Reportedly this has been recognised and ‘named’ in the organisation, but the management system has not yet fully reflected this. Whilst the model is quite innovative and maximising of technical expertise across a number of countries, with some functions devolved to PO staff in country, some issues in relation to staffing structure and the location of functions appear to merit closer attention in a new phase. Yet there are benefits to cross country functions as well, and professionals at different levels appear to be seen as open and problem-solving. Finally, the ex-pat dimension of CHAA’s staffing composition involves both strengths and challenges. It has indeed brought very high levels of expertise, as well as good abilities to operate without too much fear of repercussions in terms of local fall-out and stigma. Yet, the balance of Caribbean-to-expatriate staff may need some attention over time, in terms of CHAA’s fundamental ‘journey’ towards greater indigenisation in the region. In terms of research partnerships from within the region, the Principal Investigator at UCSF reflected that, with hindsight, “we could have added a lot with twinning partnerships with local researchers”.

5. Conclusions and lessons learned a. Conclusions on use of research, M&E and other strategic information (SI) The EC-CAP is indeed evidence based in its design and in many more specific locally tailored aspects. This is particularly so if we recognise the wider range of types of information included under SI, beyond formal research – including the contextually nuanced feedback of animators as it guides day-to-day programme implementation. Most research has come to fruition at the end of the three year funding cycle, meaning that findings have not been available to fully inform programme development. Lack of sufficient inception period compounded this problem – many SI studies were not properly started until at least a year into programme roll-out. The time lost due to set-up and staff turn-over had a major impact on the development of SI research and M&E systems – disrupting the development of coherent means of feedback and consistent lines of management within M&E components of work The research partnership with Intrahealth, which was focused on getting a particular job done over time, appears to have been fairly successful and perhaps of more practical value to the work on the ground than the research partnership with UCSF. With some exceptions, the latter did not necessarily add substantially to contextual knowledge of social and cultural settings (as related to sexual risk and so forth) although some of this may have been down to complicating factors associated with the ambitious design along with set up and turn-over challenges. CHAA could now usefully engage a more diverse range of research and SI partners, including and especially from within the region. Research could also be more practical and integrated better with M&E and organisational learning. In particular M&E in country captures a lot of important psycho-social and contextual detail pertaining to the life experiences, vulnerabilities and risks of the key populations with whom animators work. To date this important data has not been adequately utilised M&E systems nor as a source of research.

b. Conclusions on CHAA’s improving access by MARPs to HIV and AIDS services Access has radically improved for MARPs in the programme countries – and CHAA has played a unique and important role in developing and enhancing this. Work carried out to date has begun to institutionalise the idea that access for MARPs must be a key component of national HIV/AIDS strategies. This is notable in each of the countries where CHAA is operating. Whilst significant 36

steps have been taken integration of MARPs-oriented activities are nonetheless nascent within national strategies – they are not sufficiently institutionalised as yet to develop further in the absence of further capacity building and agenda development through EC-CAP. So far, the outreach has mainly provided information, support and supplies, but may need to be complemented with strategies for building social capital and addressing ‘structural obstacles’ (e.g. to accessing services but also to behavioural adaptations) for MARPs more explicitly and strategically (but based on local realities).

c. Conclusions on CHAA’s operational approach to civil society strengthening and its associated business model Among both state and non-state stakeholders in all four operational countries CHAA is highly valued for its engagement and support of communities and organisations – it plays a unique role in this regard. It is important to recognise the challenges in strengthening civil society in small Eastern Caribbean countries. CBOs addressing the needs of MARPs are small and of limited organizational capacity in all countries where CHAA operates. The growing of fledgling support groups for MARPs has been less successful than engaging with the few more established groups and it seems clear that a longterm and more intensive strategy will be needed to support groups for MARPs – including significant support in organizational capacity development and means of securing funding beyond CHAA’s onward granting. Close ‘hand-holding’ relationships with CSOs, MARP clients and national stakeholders is CHAA’s particular strength on the ground, but this is thinly stretched and not fully meeting its potentials. Organisational management structures and approaches, whilst apparently streamlined – eg. sharing functions across islands – is nevertheless ‘top heavy’ and senior staff turnover makes this approach and structure all the more challenging.

6. Recommendations Recommendations emerging from our conclusions are directed to USAID, the Caribbean HHIV&AIDS Alliance (CHAA) and the International HIV/AIDS Alliance (IHAA) respectively and these are summarised below.

a. Recommendations for USAID USAID should, first of all, be congratulated for conceiving of and then supporting the EC-CAP programme, which has clearly been pioneering and pivotal for promoting this evidence-informed approach to focused prevention work with most-at-risk populations within a sub-region with a concentrated HIV epidemic. It is exactly what this sub-region needs and which was lacking. This should be recognised, whilst further development and improvements are also called for. Overall, and taking into consideration both local/regional current realities (with its many constraints) and the history of the evolution of the response in the region (with its many dramas, confrontations and ‘blind alleys’), the evaluation team strongly believes that the only viable way to safeguard – and maximise on – the gains made through EC-CAP is to utilise the programmes built to date and develop these over a longer time-frame. Therefore, we recommend that:  USAID should invest in, utilise and expand the programmes built up in ECCAP, but: (i) over a longer time-frame and (ii) with increasing levels of resources going into country-based activities.  USAID should broaden the coverage in the sub-region with these or similar programmes, paying particular attention to patters of mobility in most-at-risk populations. Again, USAID is to be applauded for its emphasis on evidence-based programming and this has clearly had a beneficial broader influence both in programme countries and at regional levels. Given the necessity for local context relevance and specificity in the ongoing development of programmes, and given existing capacities for research, learning and evidence building in the region, a slightly more nuanced and regionally embedded approach to research and evidence may be advisable. USAID should also be congratulated for its use of the concept of ‘strategic information’, but we have found a tendency for many to reduce this idea to one of formal research studies – and to overlook the contextual relevance and strength of information that has produced through CHAA’s day-to-day activities (in the work on community animators for example). With the above in mind, we would recommend that:

 USAID should reconsider its earlier emphasis on research with major roles for northern research institutions and focus instead on building relevant research learning capacities and partnerships within the region.  Create an environment where research can be more strongly led by the needs of practitioners and where USAID’s lead partner/s in the region can draw more flexibly on a range of technical support providers and partners.  Strengthen the complementary role of qualitative and social science research aspects to strengthen work on marginalisation, sexuality and gender for example – especially as such research improves and broadens the evidence base for psycho-socially relevant and contextually effective HIV prevention.  Enhance a research approach that is not only better regionally located but also pays better attention to heterogeneity within the region – focusing on the needs and social contexts of different countries as much as the Caribbean overall. With regards to the role and contribution of civil society to the response to HIV and AIDS in the region, the evaluation team feels that USAID made the right decision to invest in this area at the outset, notwithstanding the particular challenges inherent in civil society strengthening in a dispersed region of very small countries. It is clear from the evaluation that governments alone could not and would not have ventured into this area of work. Despite being a difficult area to ‘programme’ it is indeed central to the success of EC-CAP and its objectives. Hence, we recommend that:  USAID continue support NGOs, CBOs and FBOs in the response to HIV and AIDS, because of their complementary roles which make programmes well rounded, appropriate and sensitive to local contexts.  USAID scale-up the model to more countries, and strengthen possibilities for linkages between countries and between country and civil society movements and processes at regional levels. Scaling up the model has real potential and appears all the more advisable in this highly mobile region. The big challenge in scaling up will be to heed the constant reminder from respondents that the model will need to be adapted in every country. This implies that each new country is not a simple matter of rolling out a pre-fabricated design, but investing significantly in start-up development at country level, whist ensuring that investments in existing countries continue to see these through to their full potential.

b. Recommendations for the Caribbean HIV&AIDS Alliance The approach to basing strategies on evidence is a recognised strength of CHAA’s, but there is some need to rethink what kind of evidence and ‘whose evidence counts?’ Engagement in a range of research approaches may be warranted, but the balances, priorities and approaches need to become more influenced by the countries and the communities. We recommend the following:  In strategic information and communication, focus on improving monitoring, documenting and representing the content of what CHAA does best – i.e. good community driven animator work, making more use of CHAA’s wealth of – and access to – rich qualitative information.  Orient research activities towards more operations research linked more strongly to M&E and drawing more strategically on a broader range or research partners from within the Caribbean and beyond.  Recognise the increasing capacity of the community animator’s and their potential as researchers within facilitated participatory and communitybased research designs.  Develop longer-term strategies for linking research and learning to programmatic development and improvement. CHAA’s greatest claim to legitimacy is also its greatest asset: the proven ability to empower and work with people within their communities, whilst also influencing and partnering with governments.  Build on and expand the excellent community animator programme, four ways: (i) into new and nearby countries; (ii) with increasing levels of skill in counselling and behaviour change communications; (iii) linking the work of animators more strongly to community advocacy and local CBO support group strengthening (e.g. PLHIV, MSM or low-income women’s groups) – such that skills development of animators is better linked to and institutionalised with capacity building for CBOs (iv) strengthen connections with similar programmes in larger countries in the broader region, incl. for regional referral and networking support (esp. in Jamaica, Dominican Republic and Guyana) The evaluation team are thoroughly convinced of the central and complementary importance of CHAA’s NGO/CBO support programme. It is absolutely essential in taking the response beyond a narrow MARP focus, addressing stigma and creating an environment for functional referrals and influencing society and governments at large.

 Invest seriously in developing the NGO/CBO support function at country level, by: (i) Starting with organic flexibility, but guided by strategic long-term goals, further develop systematic approaches to gradual capacity strengthening of particular high priority CBOs (ii) Extend the rationale for these partnerships to more functions beyond service delivery to include documentation, campaigning and/or advocacy – with in-country and regional training accordingly (iii) Maintain and further develop a ‘civil society sector-wide’ approach to a strategic range of partnerships, from the intensive and focused ‘handholding’ relationships to the more established NGOs and FBOs (iv) Improved and more structured (but simpler formats for) application and support procedures – e.g. include simple and clear guiding questions for applicants to address. Such procedures should also include and facilitate more rapid turn-around with processes and grant payments. CHAA has proved itself a creative and dynamic organisation, now very well positioned in a leadership role in the Eastern Caribbean. Yet, this culture of innovation must be nurtured and used to challenge the organisation itself. If CHAA moves on any of these recommendations, it is likely that the organisation will need to reconsider its own management structures, approaches and set-up more broadly. We would strongly recommend that such a review clearly engage local staff, animators and key partners across the countries, but would also like to flag a few perceived priorities and potential directions:  Protect existing local programmes and build these up further, by strengthening staffing and support at country level, empowering country office staff further, with more down-streamed decision-making abilities.  At all cost, avoid stretching country level staff thinner geographically.  Connectedly, rationalise regional level functions, within long term goals to (i) have more staff from within the region (ii) more staff from each country within programme offices (in addition to animators) (iii) rebalance the staff pyramid with a stronger base and leaner apex. This dilemma of turning the pyramid right-way up and rebalancing the ‘chiefs-toIndians’ ratio, will require serious creative thinking – e.g. out-posting different support functions to islands, or clusters of islands, streamlining the management structure, with pushing decision-making processes further down into countries etc. It is essential that staff and partners at country level are invited to freely take part and are heard clearly in any such process.  Invest further in regionalising CHAA’s image and connectedness, e.g.:

(i) by continuing to reach out to critical regional stakeholders and power brokers and proactively ‘inviting them in’ (including on the Board) (ii) enhancing country-based programme office structures through improved country-focused recruitment (iii) consider continued re-branding, but doing so with local and regional engagement in the process and outcomes

c. Recommendations for the International Alliance The Alliance has clearly had some success in the Caribbean after many years and several difficult twists and turns along the way. The decision to, and planning for, ‘indigenising’ the Caribbean office into an autonomous regional Linking Organisation (LO) appears to have strengthened CHAA’s position and potential contributions to the region over the longer term. This, of course, is exactly what the Alliance (IHAA) was originally set up to do across the world and the evaluation team have been impressed with the ‘mission development’ in this case. However, there have been difficulties (for many complex reasons, including at the IHAA secretariat itself) and, as always in these cases, much of the credit of course goes to tireless efforts of local staff and stakeholders. All of this is not to say that the Alliance has nothing left to contribute in the Easter Caribbean, the question is simply ‘what?’ The apparent disengagement in the level of technical linkages over the last few years has left a question-mark over what that future role should be in relation to CHAA, and one which goes beyond EC-CAP, but would likely impact on it. Whilst the evaluation team feels that solutions to such questions lie between the organisations involved to work out, we would flag a few recommended possibilities:  IHAA needs to engage more closely with CHAA to find out what its most useful role can be in support of CHAA - both in relation to the USAID bid as well as beyond.  IHAA will need to respect CHAA’s decisions about its own directions, but think creatively about what it (IHAA) can offer CHAA, be it in relation to broader organisational development or as a key player within the broader Alliance family and across the region (e.g. How should CHAA relate to Haiti?).  What role might IHAA play in developing and enhancing work with MARPs which has been CHAA’s ‘signature achievement’ – is there a role in facilitating linkages and learning in work with key populations in the Caribbean with the work on other IHAA LOs? CHAA is a great – and clearly hard won – asset to the IHAA family and heading to become another ‘success story’. USAID’s support has been invaluable in this and the Alliance will need to be proactive in how it engages with changing roles. 42

Annexes A1. Evaluation questions from Terms of Reference ..................................p. 44 A2. List of key informants interviewed ......................................................p. 45 A3. Questionnaires used with service clients in evaluation .....................p. 46 A4. Selected Bibliography ............................................................................p. 51 A5. Statistical tables of EC-CAP monitoring data ......................................p. 53

A1. Evaluation questions from Terms of Reference (TOR) Evaluation objective 1. Assess to what extent strategic information successfully informed interventions and influenced evidence based design? 2. Determine the projectâ&#x20AC;&#x2122;s contribution to improving the access of MARPs to HIV community services,

Specific related questions To what extent have strategic information activities (research studies, M & E) led to the design of evidence informed innovative programming? What was the role and added value of the programme partnership with the University of California in San Francisco? To what extent has the programme contributed to influencing/contributing to national priorities and strategies? To what extent have strategic information activities contributed to knowledge about the best strategies to decrease the vulnerability of MARPs in the Caribbean? From a MARPs perspective, to what extent have behaviour attitudes and practices changed positively or negatively regarding HIV/AIDS SRH and positive prevention in all four countries? What is the level of satisfaction of MARPs clients receiving community HIV services? How effectively was community based rapid testing implemented in Antigua and Barbados ? What key results were achieved? What was the role and added value of the programme partnership with Intrahealth? How effective were bi-directional referral systems implemented in Antigua and Barbados?

3. Through the use of qualitative and quantitative data, assess the programmeâ&#x20AC;&#x2122;s achievements, effectiveness and relevance, focusing particularly on the role of community (and community animator outreach workers) and civil society involvement

What were the most effective mechanisms for building capacity among the implementing partners? What key challenges were faced and how were these addressed? How is the funding of CBOs, through the small grants, contributing to the achievement of the programme objectives and results? How relevant is this approach /model to the Caribbean context? How has the programme maximised the efficiency and likely sustainability of its activities and achievements? How effective were programme management arrangements?, was the programme structure relevant and appropriate for delivery against the objectives of the programme?

A2. List of key informants interviewed 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 53. 54. 55. 56. 57. 58. 59. 60. 61.

Basil Williams Audrey Christophe Damani Goldstein Marissa Thomas Ernest Messiah Sharon Motley Donald Simeon Carl Browne Ian McKnight Angela Davies Jane Armstrong Dyllis Mc Donald Caroline Allen Andree Mc Donald Teddy Leon Donovan Emmanuel De Souzar Chery Amanda Clarke Nicole Drake Francine Bess Basil Hunte Pole-dance club owner Louise Tillotson Kezreen Ettechson Orin Jerrick Nadine Kassie Lydia Delcora Williams Reverend Karen Brotherson Lyndale Weaver Greenaway Tamara Huggins Kevin Farara Carlisle Grant Sidney Joseph Makeba Glenn Dr Del Hamilton Margarita Tash Verlene Saunders Patrick Prescod Kenyatta Lewis Serna Samuels Anita Nanton Rose Claire Charles Nadine Kassie Kevin Ramjohn Adele Henry Wilma Joseph Noel Stevens Dr Sybile Allen Jones Terry Morris Gardenia Richardson Celia Christopher Bishop John Collins Azilla Clarke Beauty shop owner Outreach client (MSM) Gaelle Bombereau Janet Myers Karen Blyth Anamaria Bejar Karine Gatellier

Executive Director (ED), CHAA Research Officer, CHAA BCC Technical Adviser, CHAA Research Assistant, CHA A Regional Director, UNIADS Caribbean Coalition of NAP Coordinators Director, Caribbean Health Research Centre Chair of PANCAP; CHAA Board-member ED, Caribbean Vulnerable Communities Cognisant Technical Officer, USAID Chief of Party, EC-CAP, CHAA Programme Director, CHAA Research Director, CHAA BCC Team, CHAA Programme Officer, CHAA CHAA Animator CHAA Animator CHAA Animator Assistant Director, National AIDS Council HIV/AIDS Coordinator, Ministry of Housing Senior Counsellor, Ministry of Health Bridgetown Programme Officer, CHAA CHAA Animator CHAA Animator CHAA Animator CHAA Animator Director National AIDS Programme Secretariat HHH Network Antigua Planned Parenthood Association Gender Affairs Programme Officer, CHAA CHAA Animator NAS Outreach worker and evaluation volunteer NAS Outreach worker and evaluation volunteer Director, National AIDS Secretariat Infections Disease Clinic, Kingstown Planned Parenthood St Vincent Population Services International, PSI Association of Evangelical Churches United Methodist Church House of Hope VCT counsellor and former NAS staff Programme Officer, CHAA Programme Assistant, CHAA CHAA Animator CHAA Animator CHAA Animator International University of Nursing CHLP Ministry of Health Director of Gender Affairs Anglican Bishop Ministry of Social Services Bassetrre Bassetrre M&E Consultant, CHAA Principal Investigator, UCSF IntraHealth International Head of Team, LAC, IHAA Programme Support Officer, LAC

Regional Trinidad Regional Trinidad Regional Trinidad Regional Trinidad Regional Trinidad Regional Trinidad Regional Trinidad Regional Guyana Regional Jamaica Region/Barbados Region/Barbados Region/Barbados Region/Barbados Region/Barbados Barbados Barbados Barbados Barbados Barbados Barbados Barbados Barbados Antigua Antigua Antigua Antigua Antigua Antigua Antigua Antigua Antigua St Vincent St Vincent St Vincent St Vincent St Vincent St Vincent St Vincent St Vincent St Vincent St Vincent St Vincent St Vincent St Kitts St Kitts St Kitts St Kitts St Kitts St Kitts St Kitts St Kitts St Kitts St Kitts St Kitts St Kitts St Kitts Canada USA USA UK UK

A3. EC-CAP Evaluation Client satisfaction interview questionnaire Interviewer/s: Location/venue: Date: Start by explaining the evaluation, your own role in it, the confidential nature of the client survey and their freedom not to take part, or to stop at any time. Confirm that this is understood (Y/tick ) First, I would like to ask a few questions about yourself: 1. How would you describe your gender? (If asked for clarification; E.g. male, female, transgender M-F/F-M, intersex, etc.) 2. How old are you (yrs)? forthcoming]

[Note:

Donâ&#x20AC;&#x2122;t

push

the

info

not

3. Which country are you from? 3.b. If from elsewhere, how long have you been here (months)? 3.c. Do you live alone or in what living arrangement do you have?

3.d What occupation/s and activities help you get by?

The rest of my questions will be about access to sexual health information, services and about the work of CHAA and itsâ&#x20AC;&#x2122; animators, as well as some other health services: 4. What do you know about CHAA (the Caribbean HIV and AIDS Alliance) and the work it does?

5. Have the CHAA activities (outreach or other services) benefited you directly in any way?

6. Do you know how to use condoms and why they are useful? 6.b. If so, where did you learn about that?

7. Have you been able to make some changes in your health, hygiene and sexual practices over time, since before you had contact with the CHAA 46

outreach workers? (Y/N/DK) 7.b. If so, what changes have they been / have you made?

8. Out of all (penetrative) sexual encounters that you have had over the last six months, can you estimate what proportion has been with using condoms? (0 – 100%) 9. Can you estimate what that proportion was approx. two years ago? 10. Do CHAA’s activities meet the specific needs of yourself and your peers?

11. Can you think of some specific aspects of CHAA’s activities which are good?

11.b. And, any which are not helpful?

11.c. How could CHAA’s services be improved for the future?

12. Overall – on a scale of 10 – how would you rate CHAA’s support / outreach? (0 = very bad; 10=Excellent) 13. Do you feel you have enough information and knowledge on sexual risk and health for the future? 13.b. If not, what more do you think you need?

13.c. Do you share knowledge you have learned from CHAA with your peers and how often?

14. Are CHAA’s animators and services easily accessible (Y/N/somewhat)? 14.b. If so, in what ways? If not, why not? (e.g. frequency, other factors)

15. On a scale of 0-10, how easily do you access other health care services?

15.b. Which other service providers have you come across?

16. Have you been referred by CHAA people to HIV testing and counselling (Y/N/DK)? 16.b. What were (on a 0 – 10 scale, poor=>excellent)

those

experiences

like?

16.d. Specifically, where they timely enough and useful enough in your view?

17. Have you been referred by CHAA animators or workers to other services? (Y/N/DK) 17.b. If so, what were (on a 0 – 10 scale; poor –excellent)

those

experiences

like?

17.c. Can you give some reasons for your rating of this?

18. How do you feel about buying your own condoms?

19. Is there anything else you want to tell us about CHAA services, outreach and other sexual health services here?

Discussion guide for outreach client 1-to-1 interviews Guidelines for interviewers: The questions in this 1-to-1 interview guide are to help you with carrying out in-depth interviews with selected key informants who you believe will be able to contribute interesting knowledge to our evaluation research. We are asking you to select people who you think will make good key informants, based on your knowledge and experience of them. We are requesting that each animator should aim to conduct 3 to 4 key informant interviews. Each interview should last for about an hour and we ask that you make detailed notes either during or immediately after the interview. This will help to capture lots of detail about what the person had to say. We suggest that notes of between 2-3 pages of A4 paper will be about the right length. Detailed notes from the interviews should be typed up and sent to Jerker and Paul by 19 th November. You are not being asked to go through all the questions given here in order. Rather this discussion guide is to help you in conducting an open ended interview with key informants. The questions are intended to help you in guiding the interview toward the key issues of the research but the discussion should be natural and conversational â&#x20AC;&#x201C; in the style that you may use during your normal outreach work. There is no need to read the exact questions to informants. Rather you can aim to include the questions in the flow of conversation. If informants bring up other issues, feel free to explore them and to ask questions that are not in the discussion guide, but which you think offer important insights and information about your work. Interviewer/s: Location/venue: Date: Start by explaining the evaluation, your own role in it, the confidential nature of the client survey and their freedom not to take part, or to stop at any time. Confirm that this is understood (Y/tick ) First, I would like to ask a few questions about yourself: 1. Can you tell me something about yourself, such as your nationality and where you come from? (E.g. other country, island etc.) - how long you have been living here and with whom (if anyone) do you live with? - how many islands/countries have you visited in the last six months? - and what work and leisure activities do you engage in? 2. Can you tell us about any health promotion/social support/HIV prevention activities or services that you have come into contact with? [Note to interviewer: We aim that this should include CHAA]

3. How did you come into contact with these activities or services?

4. How long have you been accessing or taking part in CHAA’s activities or services? - How has your contact with these activities developed over time? - Have you noticed any changes in your sexual practices and attitudes toward sexual health and hygiene? - Have you noticed any other changes in your life since coming into contact with CHAA? 5. Were these activities or services aimed at specific people such as sex workers, men who have sex with men or people living with HIV? - Did you find that such targeted services are relevant to your specific needs? 6. What have been some of the most helpful aspects of the activities or services that you have experienced? 7. What might be some of the least helpful aspects of the activities or services that you have experienced? 8. Have you been referred for HIV testing and counselling services? If yes, can you tell us about how you were referred – who gave you the information? 9. If you have attended HIV testing and counselling what were your experiences of these services – both good and bad? 10. Have you been referred to other sexual health/HIV or reproductive health services by CHAA outreach workers? - If so, which services were you referred to? - What were your experiences when attending these services? - Were they relevant to your needs (as a sex worker, man who has sex w/ men etc.)? 11. How do you see CHAA project services (incl. animator outreach and any other you have come across) as compared to other sexual health service providers? 12. Is there anything else you want to tell us that might be useful to the evaluation?

Selected Bibliography Braithwaite B. and Sex Worker Project Team (2008) Report on the Baseline Study on HIV/STI Prevention Among Sex Workers in Barbados, Govt o Barbados, HIV/AIDS Programme, Bridgetown Bombereau, G. and C. Allen (2008) Social and Cultural Factors Driving the HIV Epidemic in the Caribbean: A literature review, Caribbean Health Research Council (CHRC) CARICOM Secretariat (2009) Prostitution, Sex Work and Transactional Sex: in the English-, Duth- and French-speaking Caribbean; A literature review of definitions, laws and research, by Kampala Kempadoo, Consultant, Pan Caribbean Oartnership Against HIV and AIDS (PANCAP), Caribbean Community Secretariat:Georgetown, Guyana Caribbean HIV & AIDS Alliance (2010) Understanding Populations at Risk for HIV Infection: in St Vincent and the Grenadines: An HIV and AIDS Situation Assessment, Supported by USAID, International HIV/AIDS Alliance and University of California San Francisco (UCSF). Caribbean HIV & AIDS Alliance (2011), Gaps in Information on HIV in the Eastern Caribbean; as perceived by Stakeholder in Antigua and Barbuda, Barbados and St Kitts and Nevis, supported by USAID, International HIV/AIDS Alliance Caribbean HIV & AIDS Alliance (n.d.) Assessing the Feasibility and Acceptability of Implementing the Mpowerment Project, an Evidence-based HIV Prevention Intervention for Gay Men in Barbados, Supported by USAID, International HIV/AIDS Alliance and University of California San Francisco (UCSF). Caribbean HIV & AIDS Alliance (n.d.) Assessing the Feasibility and Acceptability of Implementing Evidence-based HIV Prevention Interventions for Women Working in Industrial Estates in St Kitts, Supported by USAID, International HIV/AIDS Alliance and University of California San Francisco. Caribbean HIV & AIDS Alliance and Community Wellness Project (2010) Sisters Infirming Sisters about Topics on AIDS (SISTA) for Caribbean Women: Draft Implementation Manual, supported by Centres for Disease Control and Prevention (Department of Health and Human Services, USA), USAID and International HIV/AIDS Alliance Caribbean HIV & AIDS Alliance and University of California San Francisco (2011) The Role of Faith-Based Organisations in HIV Prevention and Services: A situational Analysis in Antigua and Barbuda, Supported by USAID and International HIV/AIDS Alliance Caribbean HIV & AIDS Alliance and University of California San Francisco (2011b) The Role of Faith-Based Organisations in HIV Prevention and Services: A situational Analysis in Barbados, Supported by USAID and International HIV/AIDS Alliance

Caribbean HIV & AIDS Alliance and University of California San Francisco (2011c) The Role of Faith-Based Organisations in HIV Prevention and Services: A situational Analysis in St Kitts and Nevis, Supported by USAID and International HIV/AIDS Alliance Caribbean HIV & AIDS Alliance and University of California San Francisco (2011d) The Role of Faith-Based Organisations in HIV Prevention and Services: A situational Analysis in St Vincent and the Grenadines, Supported by USAID and International HIV/AIDS Alliance IntraHealth International (2010) Eastern Caribbean Community Action Program: IntraHealth International; Final Report, Unpublished report Joseph, J. and P. Faura (2008) Antigua/Barbuda (2008): HIV/AIDS TRaC Study Among Spanish-Speaking Sex Workers: First Round, Population Services International, Washington DC Kempadoo, K (2004) Sexing the Caribbean: Gender, Race and Sexual Labour, New York: Routledge Kempadoo, K (2007) ‘The War on Human Trafficking in the Caribbean’, Race and Class 49:79-84 University of California San Francisco (2010) Eastern Caribbean Community Action Program (EC-CAP) University of California San Francisco – Final Report to IHAA, University of California San Francisco (UCSF), Unpublished report.

Table A1. People reached by CHAA, Oct 2007 – Sep 2010 Indicator Number of people reached through HIV prevention activities (Males) Number of people reached through HIV prevention activities (Females) Number of people reached through HIV prevention activities (UNSTATED)

2007 Q1 M 274

2008 Q2 393

2008 Q3 431

2008 Q4 564

2008 Q5 1088

2009 Q6 2682

2009 Q7 1540

2009 Q8 2207

2009 Q9 1777

2010 Q10 1157

2010 Q11 1101

2010 Q12 948

Totals

F 434

410

330

303

450

1304

1454

1602

1185

1052

995

896

10415

U 0

158

14162

Table A2. Commodities distributed by CHAA, Oct 2007 – Sep 2010 Indicator Number of distributed

condoms

Number of lubricants distributed Number of different IEC materials developed Number of IEC events conducted Number of IEC materials disseminated

2007 Q1 14404

2008 Q2 45290

2008 Q3 30530

2008 Q4 23132

2008 Q5 117680

2009 Q6 100267

2009 Q7 207746

2009 Q8 158023

2009 Q9 154480

2010 Q10 120581

2010 Q11 189630

2010 Q12 227631

Totals

4518

8248

3494

10875

16635

15623

48900

42566

39610

29761

61610

61612

343452

382

935

1030

8749

2478

6300

14854

22937

10975

44030

62141

175746

1389394

Table A3. People referred for VCT, and people tracked as accessing counselling, testing and care, Oct 2007 â&#x20AC;&#x201C; Sep 2010

PC data

CT Data

Referrals

Indicator Number of people referred for VCT services in the reporting period (Males) Number of people referred for VCT services in the reporting period (Females) Number of individuals who received counselling and testing for HIV and received their results Number of individuals who received counselling and testing for HIV and received their results Number of individuals who received counselling and testing for HIV and received their results Number of individuals who test HIV positive Number of individuals who test HIV positive Number of individuals who test HIV positive No of indi who received Palliative care - IP No of indi who received Palliative care - IP No of indi who received Palliative care - IP

2008 Q2 77

2008 Q3 65

2008 Q4 151

2008 Q5 211

2009 Q6 693

2009 Q7 467

2009 Q8 649

2009 Q9 1253

2010 Q10 416

2010 Q11 121

2010 Q12 102

Totals

2007 Q1 16

185

137

913

1065

920

818

547

387

428

5527

121

198

203

209

339

1143

247

345

381

465

555

2073

1351

1402

163

118

229

4221

Table A3. People trained by CHAA, Oct 2007 â&#x20AC;&#x201C; Sep 2010 Indicator Number of individuals trained in strategic information (includes M&E, surveillance, and/or HMIS) Number of Service providers trained in HIV prevention (Males) Number of Service providers trained in HIV prevention (Females) No. of ind trained to provide counselling and testing according to national or international standards No. of ind trained to provide counselling and testing according to national or international standards No. of ind trained to provide counselling and testing according to national or international standards No of ind trained to provide Palliative careIP No of ind trained to provide Palliative care â&#x20AC;&#x201C; IP No of ind trained to provide Palliative careIP

2007 Q1 0

2008 Q2 0

2008 Q3 0

2008 Q4 1

2008 Q5 0

2009 Q6 5

2009 Q7 39

2009 Q8 33

2009 Q9 35

2010 Q10 0

2010 Q11 21

2010 Q12 0

Totals

M 0

132

188

127

134

Table A3. Studies conducted by CHAA, Oct 2007 â&#x20AC;&#x201C; Sep 2010 Indicator Number of conducted

Studies

2007 Q1 0

2008 Q2 0

2008 Q3 0

2008 Q4 0

2008 Q5 0

2009 Q6 0

2009 Q7 1

2009 Q8 2

2009 Q9 2

2010 Q10 1

2010 Q11 1

2010 Q12 2

Totals 9