External evaluation of SALIN+ projects Global overview report February – June 2011
Who we are
The International Planned Parenthood Federation (IPPF) is a global service provider and a leading advocate of sexual and reproductive health and rights for all. We are a worldwide movement of national organizations working with and for communities and individuals. IPPF works towards a world where women, men and young people everywhere have control over their own bodies, and therefore their destinies. A world where they are free to choose parenthood or not; free to decide how many children they will have and when; free to pursue healthy sexual lives without fear of unwanted pregnancies and sexually transmitted infections, including HIV. A world where gender or sexuality are no longer a source of inequality or stigma. We will not retreat from doing everything we can to safeguard these important choices and rights for current and future generations.
  External evaluation of SALIN+ projects: Overview report 2010 iii
Contents
Acronyms and abbreviations
iv
Foreword
v
Executive summary
7
1 Introduction to SALIN+ and the Evaluation
10
2 Method and Approach
12
3 SALIN+ country overviews and data
14
4 Findings – key achievements and outcomes
16
5 Conclusions, lessons and recommendations
38
Annexes
45
iv External evaluation of SALIN+ projects: Overview report 2010
Acronyms and abbreviations
ACPD: Albanian Center for Population and Development
NGO: Non-governmental organisation
AIDS: Acquired immunodeficiency syndrome / Acquired immune deficiency syndrome
NO: National Office
AMODEFA: Mozambican Association for Family Development APROFAM: La Asociación Pro Bienestar de la Familia de Guatemala ASRH: Adolescent Sexual & Reproductive Health ATSR: Tunisian Association of Reproductive Health BCC: Behaviour change communication CBO: Community based organisation CMIS: Client Management Information System CPD: Continuing professional development CSE: Comprehensive sexuality education EC: Emergency contraception Five ‘A’s: Abortion, Access, Adolescents, Advocacy, AIDS/HIV [IPPF strategic framework] FPAB: Family Planning Association of Bangladesh FPOP: Family Planning Organisation of the Philippines HIV: Human Immunodeficiency Virus HPV: Human papilloma virus Jovenes sin censura: Young people without censorship [Guatemalan youth organisation] IEC: Information, education and communication IPH: Institute of Public Health IPPF: International Planned Parenthood Federation LGBT: Lesbian, gay, bisexual, transgender LGU: Local Government Unit MA: Member Association MDG: Millennium Development Goals M&E: Monitoring and evaluation MOH: Ministry of Health MOHFW: Ministry of Health and Family Welfare MOU: Memorandum of understanding MR: Menstrual regulation Multiplicadores: Guatemalan term for peer educators
OSYP: Out of school young people PGB: Programa Geracao Biz (‘Biz Generation’) – National Mozambique sexual health programme for young people PLWA: Person Living with HIV/AIDS PPT: Powerpoint RH: Reproductive health SALIN+: Strategic Alliances with International NGOs SRH / R: Sexual and reproductive health / rights STI: Sexually transmitted infection TOT: Training of trainers UBR: Unite for Body Rights UNESCO: United Nations Children’s Educational Fund UNFPA: United Nations Population Fund VCAT: Values Clarification & Attitude Clarification Training WHO: World Health Organisation YAM: Youth Action Movement YF / S: Youth-friendly / services YIF: Youth Incentives Fund YP: Young people
External evaluation of SALIN+ projects: Overview report 2010 v
Foreword
The International Planned Parenthood Federation (IPPF) has a strategic focus on adolescents and young people. We focus on promoting young people’s sexual rights through the provision of sexual and reproductive health services, social and political change and access to comprehensive sexuality education. Every year we provide over thirty million sexual and reproductive health services to young people through our 155 Member Associations.
“The results of our evaluation will be used as an organisational learning tool to promote critical thinking and learning throughout the Federation in years to come. We hope this evaluation will inspire you, too”
In 2009, the Ministry of Foreign Affairs of the Government of the Netherlands awarded extra funding to IPPF for two years through the Strategic Alliance with International NGOs (SALIN) funding stream. The funding aimed at strengthening the implementation of IPPF programmes that work with and for young people, with a specific emphasis on increasing access for under-served young people and improving the quality of sexual and reproductive health services.
Doortje Braeken
Funding was awarded to eighteen Member Associations (MAs), three per IPPF region. Through these eighteen MAs, IPPF provided over 13 million services to an estimated 6.7 million young people. This represents an increase of 64% from the start of the project and a significant scaling up of service delivery and outreach operations, through which we were able to reach out to millions more young people.
We want to thank the independent evaluators who undertook this massive effort – Ms. Susie Daniel and Ms. Barbara James. Their hard work, dedication and passion for our issues was undeniable, and they were a pleasure to work with. Finally I would like to thank the Member Associations in the 6 countries that took part in the evaluation and the young people who gave their time and skills. The results of their work will inspire us all into action in the years to come.
Doortje Braeken At the end of 2010, IPPF decided to conduct an external evaluation in six of the participating Member Associations to learn from the results, challenges and opportunities provided through the extra support and funding in 2009 and 2010. The evaluation framework assessed progress not only against the four objectives of the SALIN+ Fund, but also whether or not elements essential to the provision of youth-friendly services were in place. The results of our evaluation will be used as an organisational learning tool to promote critical thinking and learning throughout the Federation in years to come. We hope this evaluation will inspire you, too, to strengthen sexual and reproductive health services for all young people.
Senior Adviser Adolescents/Young People IPPF, London
vi External evaluation of SALIN+ projects: Overview report 2010 
External evaluation of SALIN+ projects: Overview report 2010 7
Executive summary
Introduction
The Government of the Netherlands awarded funding to the International Planned Parenthood Federation (IPPF) for 20092011 to strengthen the delivery of their Adolescent ‘A’ Strategic Framework. The funds came from the SALIN+ (Strategic Alliances with International NGOs) fund to address failing efforts toward Millennium Development Goal 5 (MDG).1 The IPPF SALIN+ programme had a focus on increasing access for underserved young people and improving the quality of services for young people (see SALIN+ objectives in the table below). IPPF commissioned an external evaluation of the SALIN+ Fund. The review focused on the following universal objectives along with the elements that enable organisations to function effectively: Evidence – reports and other documentation, interviews with staff and partners, clinic visits, focus groups, interviews by young people with other young clients/non-clients – was considered under these headings and rated as the following three categories. In some cases a plus (+) or minus (-) was added to indicate that the organisation rated slightly under or over the standard rating: 1 (GS): getting started 2 (MP): making progress 3 (SP): systematically in place2
Young people were trained to carry out interviews with other young people and feedback the results at the validation workshop. Workshops with key stakeholders were held in each country to consider and validate the review findings. This overview report highlights the main findings from the six country reviews – Albania, Bangladesh, Guatemala, Mozambique, Philippines and Tunisia. Emerging from these findings, as well as from interviews with IPPF Central Office and Regional team members, are analysis and recommendations for the IPPF Central/ Regional functions.
Summary findings and conclusions
Overall, the review found that across the countries visited, the SALIN+ programme rated a solid ‘making progress’ in each of the 11 elements. Beneath this general assessment are a range of findings, with very positive progress in some areas, as well as challenges to address. While it is not always entirely meaningful to do direct comparisons among the SALIN+ programmes because they represent such distinct demographics and local priorities and needs, some broad comparisons can be made. For example, where one organisation received a 1+ in accountability and another received a 3, it is clear that in the more highly rated programme, there is a systematic approach, and in the lower-rated organisation there is much work to be done. It will be important to read
I SALIN+ universal objectives 1 2 3 4
To strengthen, increase and expand access to youth-friendly sexual and reproductive health services for young people To increase institutional commitment to youth-friendly services and young people’s sexual rights To promote the provision of comprehensive sexuality education and information to all young people To build supportive communities for the realisation of young people’s sexual rights
II Enabling environment 5 Policy & planning 6 Information & evidence base 7 Partnership & engagement
8 Capacity & resources 9 Communication
10 Service delivery 11 Accountability
Promising practice
Added value
III Crosscutting elements Sustainability
1 M DG Five is to improve maternal health globally 2. Getting started 1) There is little or nothing in place to meet the element Making progress 2) Substantial progress, but gaps remain in some areas. If arrangements are not in place, there are plans to ensure improvement systematically in place 3) Fully incorporated across the programme. While not all elements can be in place at all times, there is an overall sense that the MA is working at that level of provision and that there are systematic arrangements to ensure delivery.
8 External evaluation of SALIN+ projects: Overview report
country reports for full details. There were some universal areas of achievement, including the enhancement of clinical and outreach services, augmenting of peer education programmes and increased advocacy and partnership working. The SALIN+ programme was part of the global transition from unrestricted to restricted funding, and for many Member Associations, this was a first experience of focusing on outcomes and more extensive data collection. SALIN+ offered IPPF and the Member Associations an opportunity to strengthen and increase their work with and for young people. Despite the challenges of one-off, short term funding, all organisations have done an impressive job of developing and implementing a wide range of youth services. Some of the review findings include: Planning: Some Member Associations found the SALIN+
planning challenging, particularly in context of the short timeframes but also in terms of organisational planning skills. Information and evidence base: Most reviewed organisations initially found the requirements for the collection and analysis of data challenging. However, most reported that the additional data requirements would have longer-term benefits in helping them assess the impact of programmes. Data gaps still remain and there is not well-established use or development of evidence of effectiveness. Partnership and engagement: There were some positive examples of partnership working. The majority of these were
already in place before SALIN+ but these existing partnerships were strengthened through the SALIN+ work and partners noted this. Engagement with parents and other gatekeepers (religious and community leaders) was not developed as fully as possible. Communication: In some countries there were clear systems in place to support partnership work and share lessons. Several programmes made good use of the media to support and promote their work to the wider community and the services to young people. Communication between SALIN+ programmes, country partners, Regional and Central Office was not adequate to support effectiveness. Organisational capacity, resources and development: Member Associations that had the systems and ‘basics’ in place were able to deal better with the challenges of the SALIN+ programme. Capacity building was essential for not only the peer educators but also as a means of ensuring all staff – clinical, administrative, support – understand and accept young people’s SRH rights and needs. More performance management and quality assurance are needed to ensure effective, youthfriendly services are provided in the longer term. Sustainability and coverage: In review countries the delivery of youth-friendly services was generally increased and more diverse populations reached through outreach and improved access. In some countries with cultural and religious barriers, inconsistencies were noted in relation to ensuring universal access to young people for all services.
External evaluation of SALIN+ projects: Overview report 2010 9
The programme has initiated a large number of youth centres that are very youth-friendly, but are relatively high cost and reach limited audiences. While these centres are newsworthy and create interest in youth-friendly provision, it is also important to create a balance and ensure wider, less costly coverage of more vulnerable groups with at least essentially youth-friendly services. Most service users, especially of clinical services, were mainly female; more work is needed to reach young men. Sustainability remains the greatest challenge for all of the SALIN+ programmes – how to continue the work but with much less funding. Accountability and governance: Many countries have achieved the IPPF standard that 20% of the governing board members should be under 25 but there are issues of how meaningful the participation is.
IPPF Central Office
IPPF Central Office experienced some of the same issues that were identified in the Member Associations, such as time and capacity limitations for planning, reviewing and overseeing. Communication with Member Associations was not always timely or understood by countries. For example, IPPF has clear guidelines on use of language in IEC materials but few of the programmes visited had seen the guidelines or they were not available in the relevant languages.
Recommendations
The first recommendation that has been made in all country reports, and is repeated here, is to keep up – and build on – the good work. There are many positive developments that can point the way to improved SRH rights and access to SRH services and comprehensive sexuality education. However, to be able to do this
with reducing resources it will be important to be rigorous about identifying what the key factors for success are and to ensure the basics are in place across Central Office, Regions, Member Associations and local service providers that will allow IPPF to achieve its goals. The following crosscutting recommendation areas emerged for IPPF Central Office: planning information and evidence base partnership and engagement communication sustainability and coverage organisational development accountability
One key recommendation is to ensure a balance between relatively high cost, very youth-friendly services that serve limited numbers and wider services that could be developed to have at least essential youth-friendly provision (i.e. confidential, non-judgemental and involving young people in monitoring). Regional offices are asked to consider their role in ensuring these recommendations are carried out. Summary recommendations for member associations are in appendix 3; for details, see individual country reports.
10 External evaluation of SALIN+ projects: Overview report
1 Introduction to SALIN+ and the Evaluation The Government of the Netherlands awarded funding to the International Planned Parenthood Federation (IPPF) for 20092011 to strengthen the delivery of their Adolescent ‘A’ Strategic Framework. The funds came from the SALIN+ (Strategic Alliances with International NGOs) fund, which was developed by the Dutch Ministry of Foreign Affairs to address failing efforts toward Millennium Development Goal 5 (MDG).3 Seven international NGOs, including IPPF, received the funding to “expand access to high quality information and comprehensive reproductive health services”.4 The IPPF SALIN+ programme had a specific focus on increasing access for underserved young people and improving the quality of services for young people. Drawing on IPPF’s Adolescent Review (2007) and the IPPF Strategic Framework 2005-2012, four universal objectives for the global programme were identified: 1 To strengthen, increase and expand access to youth-friendly sexual and reproductive health services for young people 2 To increase institutional commitment to youth-friendly services and young people’s sexual rights 3 To promote the provision of comprehensive sexuality education and information to all young people 4 To build supportive communities for the realisation of young people’s sexual rights.
3 MDG Five is to improve maternal health globally 4 IPPF (22 Oct 2009) SALIN Plus Fund Handbook 2009-2010.p3
Member Associations (MAs) from all six global Regions were asked to submit proposals and three programmes from each Region were approved to receive SALIN+ funds. All of the funded projects combined innovation with a ‘getting back to basics’ approach. See the full list of SALIN+ Programmes in Appendix 1. Six programmes were identified for the evaluation, one in each Region: Albania – European Mozambique – Africa Network Region Bangladesh – South Asia Philippines – East & South Region East Asia & Oceania Guatemala – Western Region Hemisphere Region Tunisia – Arab World Region The evaluation team of Susie Daniel and Barbara James was appointed following an open tendering process. Both evaluators carried out the pilot review in Guatemala with Kat Watson (Youth Officer, IPPF London). Barbara did the reviews in Albania and Bangladesh and Susie did the reviews in Mozambique, Philippines and Tunisia.
External evaluation of SALIN+ projects: Overview report 2010 11
A SALIN+ Universal objectives1 1 To strengthen, increase and expand access to youth-friendly sexual and reproductive health services for young people
1.1 Health based outcomes 1.2 Rights based outcomes
2 To increase institutional commitment to youth-friendly services & young people’s sexual rights
2.1 Institutional commitment
3 To promote the provision of comprehensive sexuality education and information to all young people
3.1 Comprehensive sexuality education (CSE) 3.2 IEC and Peer educators
4 To build supportive communities for the realisation of young people’s sexual rights
4.1 Advocacy 4.2 Involvement of parents and other gatekeepers 4.3 Community awareness
B Enabling environment 5 Policy & planning
5.1 Policy environment 5.2 Plans
6 Information & evidence base
6.1 Needs assessment6.2 Data and information 6.3 Effective and innovative
7 Partnership & engagement
7.1 Core partnerships 7.2 Wider partnerships 7.3 Civil society engagement
8 Capacity & resources
8.1 Job descriptions & recruitment 8.2 Training 8.3 Skills 8.4 Supervision 8.5 Resources
9 Communication IEC (see above, element 3)
9.1 Partners – external 9.2 Media 9.3 National network – staff/internal
10 Service delivery
10.1 Range – of services 10.2 Range – of clientele 10.3 Accessible, YP friendly services
11 Accountability
11.1 M+E 11.2 Lesson learning 11.3 Governance
C Crosscutting elements (not rated) Sustainability Promising practice Added value
12 External evaluation of SALIN+ projects: Overview report
2 Method and Approach
that need to be in place for organisations to function effectively – ‘the why and how’. The following table summarises the elements. See appendix 3 for the full framework. The enabling environment sections aimed to look at whether there was a systematic approach to the work in the organisation that would give an indication of the sustainability of the initiatives undertaken. For example, in relation to information and evidence base, whether systems are in place for collection, analysis and use of data and information that informs effective planning, delivery and monitoring of service delivery and that evidence of effectiveness is used in planning and collected and analysed throughout the project. Accountability looks at whether systems are in place to ensure governance that includes the voice of young people. For country reports, evidence – reports and other documentation, interviews with staff and partners, clinic visits, focus groups, interviews by young people with other young clients/non-clients – was considered under the above headings and rated as the following three categories. Getting started There is little or nothing in place to meet the (1) element IPPF commissioned this evaluation of the SALIN+ programmes in the six global regions where IPPF has Member Associations. IPPF identified objectives for the evaluation, in the context of the diverse country cultural, socioeconomic and religious environments across the IPPF Regions. Although the evaluation emphasised different themes in each country, the following were the general objectives: To evaluate the Member Associations’ project against the four
global objectives (see above, introduction)
Making progress (2)
Substantial progress, but gaps remain in some areas. If arrangements are not in place, there are plans to ensure improvement
Systematically in place (3)
Fully incorporated across the programme. While not all elements can be in place at all times, there is an overall sense that the Member Association is working at that level of provision and that there are systematic arrangements to ensure delivery.
To identify and document promising practices in relation to
youth-friendly service provision and youth SRH programming that can be shared across IPPF and with external partner organisations To assess any unexpected added value or impact/results of the SALIN+ projects beyond the set goal and objectives of the projects The overall methodology aimed to be participatory and empowering to ensure the evaluation focuses on developing a consensus view of the outcomes and lessons learnt in each Member Association project to inform global findings. IPPF identified key questions in relation to each of the four global objectives. In addition to considering the four universal objectives – the ‘what’ of the review, the consultants have considered the basics
A rating of ‘getting started’ was relatively rarely given, and was considered by the organisations receiving it as an area needing serious consideration and action. A rating of ‘systematically in place’ was also relatively rare. Many of the stronger organisations felt that ratings in the ‘making progress’ range were appropriate, as there was “always room for improvement”. Initially the rating system was only the three outlined above. However as the review progressed, it became clear that most ratings were in line with a ‘bell curve’, that is, relatively few at the ‘best’ and ‘worst’ end of the scale and the majority in the ‘making progress’ category. As a result, in some cases a plus (+) or minus (-) was added to indicate that the organisation rated slightly under or over the standard rating (for example where performance was
External evaluation of SALIN+ projects: Overview report 2010 13
better than just ‘making progress’, but it was not systematic, so a 2+ would be given). For this overview report, additional interviews were undertaken with regional and central staff. Key value-related issues have informed the approach, including: 1 the Adolescent Review (2007), IPPF Mid-term Review (2010), and the Member Association SALIN+ project reports and other key IPPF documentation 2 the sexual rights framework of IPPF 3 the premise that young people are rights-holders who have the right to be actively involved in decision-making processes that impact not only on their sexual and reproductive (SRH) and wellbeing but also on other issues that affect their lives 4 a holistic view of the rights and development of young people, including health outcomes and contraceptive uptake but not exclusive of other impacts 5 empowering IPPF Member Associations (MAs) by allowing space for constructive, critical and collaborative thinking. The evaluation process aimed to help MAs to identify training needs and support on how to use the evaluation for future programme development The approach of the evaluation was based on the following values: 1 Organisational learning at all levels of IPPF – the six SALIN+ countries of the evaluation, the 18 SALIN+ country programmes and IPPF generally. The aim is to identify successes, challenges and lessons learned and the reasons for these, with an aim to find key approaches that will be relevant to ongoing project development and the overall work of IPPF. Specific areas include evidence of effectiveness, achievability and sustainability. 2 Participation of all relevant stakeholders, especially young people, throughout the process. The processes to achieve this include young people’s training and carrying out of interviews and the validation with stakeholders at the end of the evaluation. 3 Empowerment of IPPF Member Association staff and users. This is achieved through clear communication and partnership work with each Member Association evaluated, combined with lesson-learning, participation, consensus-building and raising awareness of evidence based approaches. Essential is the involvement of young people. The involvement of young people in the evaluation process was important and in each country a core group of young people was trained in interviewing techniques and processes (see appendix 6 for training framework). They carried out interviews with other young people, including both those who use the services and those
who have not accessed them. In addition, the Member Association and, in some cases relevant partners, were involved in a workshop at the end of the review to discuss and validate the findings. This process worked well in all countries and participants valued the opportunity to discuss and validate the outcomes of the review. In the final review stages, some adjustments have been made to the scoring, aiming to ensure a balanced approach across all countries. To support the review team and ensure the highest quality of work at all stages of the evaluation process, an independent expert was commissioned to do a quality assurance review of the process and selected products. The evaluation aimed to be accessible to all and not overly technical, speaking specifically to IPPF’s work with adolescents and young people, ensuring it is an organisational learning tool. The completed framework leaves each Member Association with a planning tool, identifying strengths and weaknesses and the key elements of youth-friendly service provision (details of the framework’s elements in appendix 3). Quotes in reports are not attributed to individuals or roles, as the reviewers guaranteed anonymity to participants. In some cases only one individual in a role was interviewed, so even roles are not noted.
14 External evaluation of SALIN+ projects: Overview report
3 SALIN+ country overviews and data
3.1 Country overviews
There is a separate country report for each programme and the executive summaries from each of the country reports, including ratings, are included in appendix 3. The countries represented a wide demographic spectrum, with some countries reflecting high levels of poverty and health needs to more middle-income countries with needs that are less acute. Each Member Association developed a unique programme, some building on existing youth work, others developing entirely new services and each adding a local flavour. Much of the work built on the IPPF global strategy to work with the “Five ‘A’s” – Abortion, Access, Adolescents, Advocacy and AIDS/HIV5. Below is a brief summary of the focus of the programmes of each IPPF Member Association country programme included in the review.
Albania Before SALIN+, the Albanian Centre for Population and Development (ACPD) focused mainly on women in its SRH service provision. SALIN+ enabled the organisation to extend its SRH focus to young people in schools and universities as well as building on work with Roma and Egyptian young people, and priority groups such as survivors of domestic violence and trafficking. A central focus of the programme was to establish a new centre and clinic
for Tirana, the capital of Albania, as well as refurbishing centres in Vlora (south) and Shkodra (north). Other successes included development of a peer education programme, national advocacy for comprehensive sexuality education and strong partnerships for joined-up service provision for young people.
Bangladesh The Family Planning Association of Bangladesh (FPAB) is a large national organisation working across Bangladesh for nearly 60 years. SALIN+ enabled the organisation to build on earlier work of the Youth Incentive Fund’s youth corners to establish 10 young people’s centres – tarar melas6. These centres are a new approach in Bangladesh and allow young people to come together for a range of activities – such as training on computers and English language, competitions – as well as providing access directly to SRH clinics and to information on SRH. These youth centres have been very well received by the towns and cities where they are located. FPAB has developed productive partnerships with government and NGOs and positive engagement with parents at local level. Young people have been involved in the project as peer educators, researchers and programme reviewers.
Table 1: Some key demographics of countries reviewed7
Population (in the world)
Human development Index
Median age (years)
Rate of HIV infection (%)
Literacy (%)
2.99 million (136th in world)
64
30.4
0.8
73.8
Bangladesh
160 million (7th in world)
129
23.3
<0.1
47.95
Guatemala
14 million (69th in world)
116
20
0.8
69.1
Mozambique
21.4 million (51st in world)
175
16.8
11.5
47.8
Philippines
101.8 million (12th in world)
97
22.9
<0.1
92.6
Tunisia
10.6 million (78th in world)
81
30
<0.1
74.3
Albania
5 http://www.ippf.org/en/What-we-do/ 6 Gathering of stars 7 http://hdr.undp.org/en/statistics and https://www.cia.gov/library/publications/the-world-factbook
External evaluation of SALIN+ projects: Overview report 2010 15
Guatemala APROFAM (La Asociación Pro Bienestar de la Familia de Guatemala8) has a long history of working with young people. SALIN+ – called the ‘Mesoamerica Programme’ in Guatemala – enabled APROFAM to focus particularly on advocacy and development of its work with peer educators and enhance the clinic service provision. Young peer educators were involved in outreach and peer education as well as being recognised in APROFAM and by partner organisations as having been the key success factor in advocacy initiatives to change national policy on comprehensive sexuality education. Young people have also been involved in monitoring and evaluation exercises, which have demonstrated the successes the Mesoamerica Programme has achieved in its SRH rights-based work and service provision for young people.
Mozambique AMODEFA (Associação Moçambicana para Desenvolvimento da Família9) has a long-term partnership with the government programme ‘Biz Generation’10 for the provision of outreach work, sex education and youth-friendly services. The SALIN+ programme enabled them to target specific groups, including young people who are deaf,11 affected by and/or living with HIV/AIDS and those living in very remote communities. AMODEFA enhanced its clinical services in Maputo and Namaacha and opened new clinic and outreach services for young people in Zambezia. The peer educators were involved in most aspects of the SALIN+ programme (including in the clinics) and were integral in strengthening local and national partnerships and raising AMODEFA’s profile as a young people’s provider.
Philippines Family Planning Organisation of the Philippines (FPOP) had never previously focused on young people. Through SALIN+ FPOP established targeted initiatives for young people in four centres across the country and developed a training programme for youth volunteers, staff and board members. The reputation of FPOP has been enhanced, particularly as a provider of SRH services for young people. The centres were improved to ensure they are youthfriendly and in Pampanga they successfully targeted outreach work with young sex workers. The peer educators carried out high profile advocacy for the pending Reproductive Health Bill, with a focus on young people’s sexual health rights. A successful partnership with a college of nursing enabled nursing students to
have a practical learning experience in relation to young people’s health and SRH rights.
Tunisia L’Association Tunisienne de la Santé de Reproduction12 (ATSR) had not previously worked in a targeted way with under-25 year olds – the average age of marriage is late 20s or early 30s and ATSR’s focus had been to meet the needs of that age group. Tunisia has had a governmental family planning programme (free to all) since 1965 and legal abortion since 1973. As part of their SALIN+ programme ATSR were ambitious, enhancing youth provision at seven clinics, as well as opening three new youth centres. Due to the political turmoil, over-extension of the programme and other issues, the new centres were only opened for a short period of the programme time. In Medenine ATSR developed delivered targeted work with deaf young people and involved parents in the community outreach work..
3.2 Data
Analysis of country project data appears in the country reports and will be carried out in collaboration with IPPF Regional Offices and Member Associations for the purposes of planning for future programmes. Both increases and decreases in service delivery in such a short project must be viewed with caution as it is longer term trends that will demonstrate sustainable approaches to young people’s SRH service provision. It will be important to ensure data collection achieves a balance between basic data that is not sufficient to be meaningful and a burdensome collection of information that is not used. As IPPF moves more towards more restricted funding, it will be vital to collect sex disaggregated data on client visits, repeat service use, etc. that allows a rich analysis of effective – and cost effective – approaches.
8 Translation: Guatemala Association for Wellbeing of the Family 9 Translation: Mozambican Association for Family Development 10 Programa Geração Biz (‘Busy/Biz Generation)(PGB) is a Government programme running since 1999, now operating in all 11 Mozambican provinces, focusing on schoolbased work with counselling corners developed with the Ministry of Education; a community-based programme with youth centres under the Ministry of Sport and Youth; and youth-friendly clinic services based in Ministry of Health facilities. 11 Throughout the report ‘deaf’ is used to include all people with any form of hearing impairment. 12 Translation: Tunisian Association for Reproductive Health
16 External evaluation of SALIN+ projects: Overview report
4 Findings – key achievements and outcomes A SALIN+ Universal objectives 1 To strengthen, increase and expand access to youth-friendly sexual and reproductive health services for young people Summary In all of the countries reviewed there was notable strengthening, increase and expansion of the youth-friendly SRH services – clinical and with peer educators. It is likely that the SALIN+ programme contributed to IPPF’s reported overall doubling globally of use by services of young people under – 25s. Across programmes there have been positive achievements among underserved groups, although there has been little evidence of a targeted, needs-based approach. All of the Member Associations prior to SALIN+ offered a broad range of services but most have developed services that ensured a more youth-friendly focus and made young people’s access easier. The majority of the programmes carried out refurbishment, or developed new centres, to enhance the centres and make them more youth-friendly. For most programmes the development and enhancement of non-clinical services – outreach, community education work, etc. – has been an important element in reaching more young people. There are some examples of young people’s involvement in both the design and delivery of the programmes though for most SALIN+ programmes this was not a focus of the work. Despite a lack of in-depth needs assessment by all Member Associations, achievements do include successful work with groups of underserved young people. The young people who were involved brought energy, enthusiasm and other young people into all elements of the SALIN+ programmes. The major challenges included the lack of time to carry out needs assessments, systematically plan the work programme and identify key marginalised groups for targeted work. There was little focus on gender and the needs of young men were not specifically addressed by any of the programmes. In some countries the dominant cultural and religious values were not supportive of the promotion of young people’s SRH rights and/or services.
1.1 Health based outcomes: Achievements and successes In terms of health outcomes Tthe key health outcome looked at considered in the review was the increase in numbers of young people able to receive a range of SRH services. In addition, health
outcomes included success in reaching under-served young people, an increase in the range of services available to young people and the improvement of both clinical and non-clinical services available to young people. Overall, during the SALIN+ period, IPPF data show that there has been a virtual doubling of all service use globally. While it is likely that not all of this increase can be attributed to the SALIN+ programme, it appears to have made a contribution in reaching more young people, increasing service provision and improving the youth-friendly focus of the services. (See appendix 4 for project data). It should be noted that some Member Associations previously provided services to young people. In addition, a number of other countries programmes with a youth focus were in place, either before or alongside SALIN+. In Bangladesh, for example, the Youth Incentives Fund (YIF) initiated ‘corners’ for young people that were the forerunners of the current young people’s centres. In Mozambique there was a long history of the Member Association providing services as part of the government young people’s sexual health programme, including outreach, sexuality education and clinical services. SALIN+ enabled AMODEFA to greatly enhance their work in a wider range of settings.
“This is the first time we’ve been able to work with the underserved youth... the work we have done with the vulnerable groups is our biggest achievement – we’ve realised that some young people need extra support – they are so much more vulnerable. We’ve learned a lot.” Tunisia All countries in the review either refurbished existing centres or opened entirely new centres, and in some cases, did both. Some of the programmes used free service provision as a means of supporting young people to access the services. In the majority of the review countries, new groups of underserved young people have been successfully reached – with a strong focus across the programmes on young people in schools and universities although some countries worked with very marginalised groups of young people. In many instances, although there is not a wider range of services, the ones that the Member Association delivers are reaching more young people. For more detail of services delivered and priority groups reached, see element 10.1 and 10.2 below. There has been a marked increase in non-clinical services for the majority of the Member Associations, including outreach to
External evaluation of SALIN+ projects: Overview report 2010 17
underserved groups, youth and peer counselling, and provision of information on SRH. Outreach was the main focus of the nonclinical work, with a range of approaches – SRH education sessions with groups of young people in schools and community groups; door-to-door visits to promote the clinic services; information kiosks in town squares; use of mime, theatre and drama; mobile vehicles with clinical and/or information and promotional materials, etc. In Mozambique and the Philippines peer educators worked in the clinics welcoming and counselling young people, easing their entry into the clinic – particularly on the first visit. In Albania SALIN+ funding was used to refurbish existing clinics and acquire and develop a new centre and clinic in Tirana, the capital of Albania. While the centre has been open only for eight months and full service uptake is yet to be realised, the centre is owned by ACPD and represents a significant increase in potential coverage for the country. In Bangladesh, FPAB, which had not previously specifically targeted unmarried young people (particularly in schools and universities) developed ten centres for young people – tarar melas – youth centres12, most of which were connected directly to clinics to allow young people confidential access to clinical services. Guatemala also focused the majority of its efforts on young people in schools and universities, as well as enhancing the youth centres to provide young people-only services. Examples of reaching particularly underserved youth include two countries that did targeted work with deaf young people. In Mozambique the majority of the peer educators, as well as some clinical and administrative staff, were trained in sign language. Deaf young people became peer educators, carried out outreach work to other deaf young people and participated in all promotional activities. The training also meant that in the clinics hearingimpaired young people could receive a more confidential service from staff who could communicate directly with them without a signer from outside. In Medenine (Tunisia), partnership work was developed in a local school for deaf young people, some SALIN+ staff also learned sign language and some deaf young people were involved in outreach work.
Challenges The biggest challenge facing all of the SALIN+ programmes is sustainability and the ability to maintain the services and focus of the work with less funding, especially in centres where vouchers and free services were provided. There are additional problems for the new SALIN+ centres developed for the continuation of sufficient funding to pay running and staff costs.
Overall there was a lack of clarity about specific targets groups to ensure organisations were reaching the most marginalised and under-served young people. This is hard to achieve without indepth, ongoing needs assessment. On a global basis SRH services have traditionally focused on work with women and it is a major challenge to ensure services welcome and reach out to young men.
“With ‘Show Biz’ we have gone into remote villages and get 80 or sometimes 300 or 400 people – we do dance or drama or sing… do poems. At the same time we do [outreach events] and provide information – use the sound system to draw people in. It really works – we don’t just reach young people, the numbers of adults coming into the services has also increased.” Mozambique In Albania, further challenges include a lack of sustainable resources to address issues such as limited clinic hours – and in the Shkodra centre, limited clinic size – that prevent the service from reaching the numbers ACPD would like with clinical services. In Bangladesh, non-SALIN+ FPAB services have not yet taken on youth-friendly values, which means that many young people do not receive youth-friendly services in FPAB-served areas. In Guatemala there is a similar situation, where there is not consistency of provision across SALIN+ and non-SALIN+ APROFAM services, and even in some SALIN+ services not all services are provided to young people (see section 10).
1.2 Rights based outcomes Achievements and successes Sexual and reproductive health rights are fundamental, and a key aim of SALIN+ was to increase young people’s SRH rights. All of the programmes included a strong focus on this, often in quite different ways. In a number of SALIN+ programmes, peer educators used outreach and events to raise awareness of young people’s SRH rights. For some Member Associations, training was a key focus to increase awareness of young people’s rights but some of the training (see section 8.2) focused on information giving with less of a rights-based approach. The involvement of, young people in the design, delivery and monitoring the project was not universal, with some centres
13 Translation: ‘Gathering of stars’
18 External evaluation of SALIN+ projects: Overview report
involving them only in outreach and educational work and others involving them in planning and other service delivery. As noted above, targeted work to address gender issues was limited and working with underserved and marginalised groups of young people on rights was different depending on the priority groups for each organisation as well as factors such as previous working relationships with relevant NGOs-dependent. In most programmes peer educators acted as a key route for new young people accessing the services by promoting the services in their outreach work, among their friends and in their families and communities.
“As deaf young people we have real problems in talking about sex and sexual health because there aren’t enough words in sign language for us to use – its really hard talking about it and getting the information we need.” Tunisia There were some notable achievements in meeting the needs of underserved young people – sex workers in the Philippines, rural young people in Mozambique, deaf young people in Mozambique and Tunisia, survivors of trafficking and domestic violence and young people working in factories in Albania, homeless young people in Guatemala Near Pampanga (Philippines) is an area called ‘sex city’, a legacy of a military presence after the war. Young women from all over the Philippines come to this area to earn money in the sex industry and those who are not legally registered have problems accessing state health services. The SALIN+ youth clinic developed extensive outreach work in the streets, bars and clubs and supported all of the sex workers into the new clinic services. FPOP have an agreement with the Health Department to carry out the legal health checks for the young women (a requirement for those wishing to become legal sex workers) and continuing to offer the young women a wide range of support services. All of the programmes found it a challenge to increase the numbers of young men accessing clinical services. There were a few examples of targeted work to increase young men: in the Philippines FPOP provided circumcision in the youth centre and an unexpected outcome was that some of the young men became involved in SALIN+ and many promoted the services to their friends. For outreach work there were distinct profiles in countries – in some Tunisian centres ATSR struggled to get young women
into services, while in other centres it was mostly young women who accessed them. In Mozambique, young women in the AMODEFA centre in Maputo established a ‘Women’s Caucus’ to enable them to address the gender issues they faced daily but there were not the resources to respond to young women in Namaacha for a similar group there. The Women’s Caucus made a strategic decision to work with young men and include young men in the work they carried out, focusing on women’s empowerment and their SRH rights. In Bangladesh, one way in which the project tackled gender issues was to bring together young people of both sexes in the youth centres. This is a new approach in Bangladesh, and one that needed considerable awareness raising and advocacy with local parents. However, this has been achieved, and parents and young people are both pleased with the arrangements and feel that it has tackled some gender issues in a positive way. An example of young clients recommending the Member Association’s services, in Guatemala, exit surveys as well as interviews undertaken by young people for the review found that the vast majority of those using services would recommend them to their friends.
Challenges Some of the challenges are linked to the lack of time available for planning and in-depth local needs assessment. This also meant that there was a lack of clarity of specific target groups to develop work with. Gender equity was a key issue, with limited success in increasing the numbers of young men into the services. As commented by staff in AMODEFA, “young men just came in for condoms – we didn’t succeed in holding onto them.” In all of the clinic services young women continued to be the largest number of users. Some of the Member Associations initially found the data collection processes challenging as it was quite new to them and staff lacked the skills to analyse and use the data to identify specific target groups of underserved young people for ongoing work (see 6.2 below). In some countries there widespread were cultural and religious values that are not in agreement with the promotion of young people’s sexual health rights. For example, in the Philippines, Tunisia and Bangladesh cultural and religious values made the work of the staff and peer educators much harder and challenging. As noted above and in element 10.2, most programmes made a pragmatic – and wise – decision to focus on young people in schools and universities. However, this focus meant that many of IPPF’s priority communities did not benefit from SALIN+, even via partner organisations. Few of the programmes were successful
External evaluation of SALIN+ projects: Overview report 2010 19
in targeting some of the harder to reach groups and there was minimal work with lesbian, gay, bisexual and transgender (LGBT) young people, or homeless, sex workers, those with drug/alcohol issues and in some countries out-of-school and very rural young people. In Mozambique and Tunisia there were delays in starting services due to bureaucratic blocks – procurement and permission for building works.
2 To increase institutional commitment to youth-friendly services and young people’s sexual rights Summary In the majority of the programmes reviewed, there was notably increased organisational commitment to addressing the barriers for young people to realise their SRH rights. Although working in different environments, each programme still achieved successes in raising the profile of their services, the organisation and their work with peer educators. Through internal policy development and external promotion of young people’s SRH rights, Member Associations have successfully addressed key aspects to ensure young people’s SRH rights are met. The main challenge is to sustain the gains with reduced funds and less ability to maintain training, policy development and implementation and quality assurance.
2.1 Institutional commitment Achievements and successes The majority of the Member Associations demonstrated commitment to the development and provision of young people’s sexual health services and their SRH rights. Staff in the SALIN+ components of review organisations all demonstrated knowledge and understanding of the barriers and issues facing young people in relation to assuring their SRH rights and the commitment to address these issues in the programme work. There was positive support from the governing body and executive level in most SALIN+ Member Associations, although some resistance in parts of a couple governing boards. The majority of the Member Associations have made known that they provide youth-friendly services as a result of the new programmes and services delivered under SALIN+ and have enhanced their reputations in this way. Training, including values clarification and awareness raising, was a key aspect of the work, enabling staff to gain greater skills to develop and deliver appropriately youth friendly services.
All of the Member Associations have developed child protection and youth policies and some have implemented the policies, with training for all staff. In a minority of countries training was offered to organisation staff in all programmes and to all Member Association staff (paid and unpaid). When training was offered to all staff, it enabled greater personal and institutional changes in knowledge and understanding of young people’s SRH rights (see section 8.2 training). Most centres commented on how much they had gained from the active involvement of young people, bringing a positive energy and new skills to the organisation. FPOP was struggling as an organisation before SALIN+, in a Roman Catholic country where SRH is very controversial for all and especially for young people. SALIN+ gave a new focus and energy to the work and has helped to put FPOP back ‘on the map’. There was clear strategic leadership and commitment from the FPOP board, administrative staff and service providers for the development of youth-friendly services. The reputation of FPOP as a provider of youth-friendly services has been greatly enhanced and they are now recognised as one of the only youth-friendly service providers in the Philippines. In smaller organisations such as Albania, where there is a total of 19 staff, it was possible to provide training and values clarification to all staff. This was followed up by supervision that considered provision of YF services and increasing numbers of young people.
“Young people are AMODEFA and AMODEFA is young people – we are totally committed to our work with them” Mozambique Both Mozambique and Guatemala had well established youth services before SALIN+ and they were able to raise organisational commitment (through training) and augment their reputation through work with marginalised groups, including deaf young people and those living in remote rural areas.
20 External evaluation of SALIN+ projects: Overview report
“We need to keep this going – there is a big gap in Tunisia – have uncovered so much need with the revolution – huge disparities and so many young people not getting any services.” Tunisia Challenges As with most of the youth-friendly service and organisational issues discussed in this report, sustainability is an issue. Maintaining and increasing the institutional commitment is difficult when there is limited funding for ongoing training and performance management of youth-friendly issues and where there is loss of staff and high staff turnover. The Member Associations where all staff were trained have the potential for a greater legacy of institutional commitment but those that trained only clinical staff or only the staff who were part of the SALIN+ programme may struggle to maintain even the levels of institutional commitment they have achieved. Training (also see element 8), without systems to ensure staff, board members and service providers have changed their attitudes will not be guaranteed of success. Systems include supervision of staff in context of their attitudes to young people’s SRH rights and the provision of youth-friendly services, an effective complaints and comments system that considers SRH rights and youthfriendly services, and monitoring and evaluation systems that take youth-friendly provision into account. Some countries face cultural and religious challenges and barriers, as well as some personal resistance to change, that impact on their work, both internally and externally. It is often challenging to ensure consistent approach, attitudes and service provision. In Bangladesh, while there was demonstrable institutional commitment on the part of both staff and service providers who were part of the SALIN+ programme, this was less obvious among other staff and service providers who had not all benefited from training. In addition, some board members serving at both local and national levels not only did not support young people’s SRH rights, but also strongly and actively opposed them. In the Philippines there were major issues and challenges in openly promoting young people’s rights to SRH information and services.
3 To promote the provision of comprehensive sexuality education and information to all young people Summary In none of the countries reviewed was comprehensive sexuality education a part of the national curriculum or agreed national policy. Where sex education was taught in schools, it was usually part of the science curriculum, and did not cover issues such as emotions, sexuality and relationships. In several countries advocacy to change this situation was a strong focus of SALIN+, such as in Guatemala, where the law has been changed and Albania, where the change in the legal situation is nearly complete. Most of the SALIN+ programmes have worked in schools and/or with community-based outreach work at local level. Peer education was a core component of all of the SALIN+ work in the majority of countries, although with a mixed impact and approach. One of the biggest challenges has been successful advocacy for a nationally agreed curriculum for CSE, especially in countries with religious and cultural barriers. There were some real barriers to providing quality and appropriate training, including: the lack of a globally agreed training programme for peer educators to ensure consistent standards and quality, and the lack of locally produced IEC materials. .
3.1 Comprehensive sexuality education (CSE) “We have been able to create some changes in attitudes about CSE but there will definitely be opposition even if the RH bill is passed. The majority of the schools are owned by the church and they will oppose the implementation.” Philippines Achievements and successes In several countries there were positive examples of developing and delivering sexuality education in both schools and communities, which has been developed, despite a lack of legal context for work in schools. All Member Associations recognise the importance of a legal framework for the provision of a nationally agreed curriculum but for many this is not a reality and there are many socio-cultural and religious barriers. All programmes have reached many young people with outreach work – in both schools and communitybased situations – to inform them about their SRH rights, access to services and awareness raising.
External evaluation of SALIN+ projects: Overview report 2010 21
The peer educators in Guatemala advocated for a comprehensive sexuality programme in schools. Together with other NGOs and in partnership with the Ministry of Education, APROFAM peer educators successfully lobbied for the new policy on sex education. Their achievement was highlighted when they gave a presentation at the National Congress on International Women’s Day. In Tunisia, Bangladesh and the Philippines there are major religious and socio-cultural barriers to the provision of CSE. In the Philippines there is a pending Reproductive Health (RH) Bill but CSE is not included within the bill. As staff said ‘one step at a time’. In Albania, ACPD had a strong advocacy programme to make CSE part of national policy. The programme staff and peer educators also had a large programme of work with teachers and doctors in schools to provide information and sex education in schools. This was carried out in all three SALIN+ cities, but in Tirana, there was a pilot initiative in schools through the school doctors that allowed staff and peer educators access to students for education, including sex education topics. In Bangladesh teachers told the reviewer that peer educators’ provision of sex education in local schools was well received and valued. In Mozambique there was no law relating to sex education but AMODEFA and partners are currently advocating for the provision of a nationally agreed curriculum. The Ministry of Education is a partner of the ‘Biz Generation’ programme and there are good working relationships with many schools. The peer educators who attend these schools are supported to provide both SRH education sessions and counselling in the ‘counselling corners’ located in the schools. The peer educators are supported by a nominated teacher.
Challenges The major challenge in relation to CSE is the overall lack of government support for the development and delivery of quality sexuality education, particularly in schools. Socio-cultural beliefs were also significant barriers to this work being delivered at all. In all countries progress without a nationally agreed CSE policy, there is limited potential for local successes in CSE delivery. Without a policy framework, the development of a high quality curriculum, staff training (teachers as well as health workers) and a delivery plan for the programme to be rolled out across the country is impossible. To achieve this requires close partnerships with the Ministries of Education and Health. Not all review countries (for example, Bangladesh and Tunisia) prioritised advocacy at a national level to achieve policy change and therefore will have limited success in this area in the longer term.
3.2 Information, education and communication (IEC) / Peer educators “I have discovered the spirit of the peer educators – without much money, they accomplish a huge amount. They rise to the challenge of working with young people who may be shy and reluctant to open up… they are really good at it.” Philippines Achievements and successes Peer education was prioritised by most countries. Peer educator training and supervision were key to ensuring that the peer educators were appropriately equipped and confident in carrying out CSE in schools and communities and there were some quality examples of training programmes. However, quality assurance and sustainability of peer education programmes are yet to be addressed. The initiation or enhancement of peer education work was a focus in almost all of the programmes, some starting from scratch, others building on strong, historical foundations. In some countries the peer educators were involved in the development and delivery of the programme, in others they were involved only in only some of the delivery work. As noted in element 11 (Accountability), some of the Member Associations have achieved the IPPF aim that all governing boards are composed of 20% people under the age of 25 (although not always with a gender balance). In many places ensuring a gender balance was a challenge – in some peer programmes young women were the largest groups while in others there were more young men. In Guatemala there was an agreed peer education training programme. The programme included basic training, followed by monthly training sessions over six months and followed by an advanced training programme. New peer educators always went out with experienced ones to build up their confidence and skills. During their whole time volunteering as peer educators, ongoing training was offered. In Tunisia peer educators did outreach work in the universities but did not always have the permission of the university authorities and so they were limited in what they could achieve. In the Philippines the programme concentrated on training the peer educators and the majority of the work they carried out was outreach to communities.
22 External evaluation of SALIN+ projects: Overview report
While production of youth-friendly IEC materials was not a major priority across review countries, there were some examples of innovative materials to attract young people’s interest and draw them to sources of information, such as the larger than life sized figures of peer educators used in outreach in Guatemala and the use of Facebook to reach young people in the three centres in Albania. Young people in Albania contributed to the development of television and radio programmes on SRH issues, in which they also appeared. In Tunisia one centre developed a ‘SRHR monopoly game’ that was successfully used in working with young people. There were many examples of dynamic, vibrant peer educators making a difference in the review countries. Guatemala’s peer educators were credited with achieving policy change; in Bangladesh peer educators and youth researchers made contact with hundreds of young people, in schools, universities, the youth centres and in outreach. In Albania as well, the advocates made inroads into university campuses and schools, factories and priority communities such as Roma and Egyptian communities, providing advice, education and information. In the remote areas of Mozambique, peer educators wore SALIN+ t-shirts and rode SALIN+ bicycles. Local people began to recognise their t-shirts and the fact that they carried condoms, and so the peer educators ended up distributing condoms to both young people and adults.
Challenges The development and production of materials for information, education and communication work did not have a high focus across the SALIN+ countries reviewed. Materials were produced as part of promoting the services available but examples of the range and depth of youth-friendly educational and informational resources were limited. For all of the Member Associations, major challenges included the sustainable recruitment, training, support, quality assurance and retention of a corps of peer educators. There is a lack of an accessible (e.g. in local languages) peer education / training framework /and programme that is universally recognised and used by Member Associations. Some programmes used ‘It’s All One Curriculum’14 but others had a less structured approach. There is a need for a standardised process for the recruitment, training (initial and continuing), and the support, quality assurance and supervision of peer educators. Support and supervision are not systematically built into the peer education processes and programme, which can be problematic, particularly as peer educators may be working in unknown and difficult environments.
“The success of our programme didn’t exist before Jóvenes sin Censura [Young People without Censorship] was established to ensure a youth voice.” Guatemala Retention of peer educators is a major challenge for most programmes – young people get older, move on, get jobs, leave school/university, etc. There were no specific examples of programmes successfully addressing of this issue, and it was a concern for the majority of the programmes. Towards the end of SALIN+ Tunisia developed a partnership with a local training college (carpenters, electricians, etc.) in Monastir but it is too soon to know if these young people will stay any longer than the university students. In Bangladesh, when the programme funding ended early, peer educators had to be cut and the salaries of youth counsellors were halved.
“Jóvenes sin Censura is now taken seriously in the media. Lots of young people have been empowered.” Guatemala
4 To build supportive communities for the realisation of young people’s sexual rights Summary In some of the programmes there has been a strong focus on advocacy and involving young people in this work; others have explored or developed advocacy without involving young people in a significant way, or on a limited basis. Some of the Member Associations did limited advocacy focused at a local level. The involvement of parents and other gatekeepers was a major focus in a few programmes but in the majority of cases was more on an ad hoc basis. Overall there is a need to strengthen the engagement of parents and other gatekeepers with targeted work to raise their awareness and engage them to support young people’s SRH rights and access to services. Raising community awareness was a major challenge, with many countries facing socio-cultural and religious barriers but there were examples of positive developments
14 A programme of the key elements for the development of a rights-based, gender-sensitive and participatory training curriculum, developed by IPPF and the Population Council. Currently it is available only in English and French. http://www.ippf.org/en/Resources/Guides-toolkits/Its+all+one.htm
External evaluation of SALIN+ projects: Overview report 2010 23
4.1 Advocacy: Achievements and successes There are several examples of important successes during SALIN+, where advocacy was focused on clear outcomes and linked to partnerships for long-term success. As discussed above (3.1) in Guatemala the role of young people in the successful advocacy for the sex education bill was critical. They established an advocacy group ‘Jóvenes sin Censura’15 who coordinated and led on the work and developed strong links with the Ministry of Education and other key NGOs.
“The young people were really instrumental in one Barangay [local government]. The local leaders were proposing an ordinance to oppose everything in the RH Bill. So the young people developed a campaign on why the bill is important and they really confronted the lies being told.” Philippines NB: In early April 2011, the Government did outlaw any local ordinances against the RH Bill In the Philippines16 for many years key organisations and politicians have been advocating for a reproductive health (RH) bill. During SALIN+ a bill was pending and the youth volunteers took a proactive role, going into communities and holding public discussions about the benefits of the bill. They also carried out advocacy with local government health and education officials and despite many barriers – and some hostility – along the way, they persevered with the work. In Albania, there is a strong advocacy unit in ACPD, and they have focused on changing policy in relation to comprehensive sexuality education and feel they are close to success. Young people have been trained in advocacy, but have not yet been involved in a significant way in the initiatives. This builds on ACPD’s wider advocacy reputation, for example, in successfully getting more women in national government.
Challenges: see end of section.
4.2 Involvement of parents and other gatekeepers Achievements and successes There are limited examples of systematic engagement of parents and other gatekeepers in the SALIN+ review countries. There were examples of engagement in most countries but most of these examples were by chance and not part of a strategic approach. When parents were involved, they usually changed from not being certain about the work to fully supporting the outreach work and the need for clinical services. Often engagement with gatekeepers was as an unexpected outcome of other work. For example, some young people experienced initial resistance from their parents when they decided to become a peer educator but were often dispelled by the positive changes they saw in their son or daughter. Engagement with religious leaders was limited (apart from in Bangladesh) but there were positive examples of working with local community leaders. In the Philippines the local government chief provided premises for the clinic, enabled access into local schools for the peer educators and protected the work with the sex workers. In Bangladesh there was a well-developed, planned approach to ensuring parents and local teachers were supportive of young people’s SRH rights. Both young people and parents spoke about parents’ initial strong resistance to their children attending the youth centres or being part of the peer programme. However, the benefits of the programme in terms of the young people’s increased knowledge and confidence, and the care with which it was presented to young people, their parents and teachers meant there was ultimately very good acceptance of the programme. In addition, the parents had a very pragmatic approach. Although they did not necessarily want their children to be having sex, the view of all parents interviewed was strongly in favour of unmarried young people – even their own – having access not only to information on SRH, but also services. They did not want their children’s lives to be “ruined” by an “accident”. In Albania there is a programme of training gatekeepers using Speakeasy17, which reached many teachers (some of whom were also parents) as well as a number of parents. The programme was very well received and review interviewees spoke about the need for the programme to be further rolled out. In Mozambique, in the outreach work in remote areas, AMODEFA staff and peer educators were surprised at how much adults were attracted in to services when outreach work with a sound
15 Young People without Censorship 16 In the Philippines a ‘Barangay’ is the smallest, administrative decision-making level (village, district or ward) of local government 17 Resources developed by fpa in the UK to support parents, teachers and others to talk confidently with young people about SRH issues.
24 External evaluation of SALIN+ projects: Overview report
system was done. Both staff and peer educators noted that even the taboo of ‘family planning’ has been affected as a result of SALIN+ activities, and now women do not have to hide their use of contraception from their husbands; they are even supported by their husbands to attend the clinic. In addition the parents of the peer educators became more supportive, some have become involved and many have accessed services for themselves. In Medenine (Tunisia), the father of one of the peer educators is a poet and as a result of his son’s involvement with the peer educators, he has written some poems about SRH that are now being used in the training programme.
Challenges: see end of section
4.3 Community awareness Achievements, successes and challenges: Community awareness has not been a particular focus of most of the SALIN+ programmes, although as noted above, Bangladesh focused the majority of its efforts on awareness raising at a local level and has been very successful in doing so. All interviewees across a range of sectors were aware of and extremely positive about the youth centres. In most of the countries reviewed there are cultural, social and religious barriers that act as a barrier to achieving young people’s sexual and reproductive rights. Apart from Bangladesh there no strong examples of a systematic approach to the engagement of a range of key stakeholders in communities to address these barriers, including faith bodies, local community leaders, parents, (as above), etc.
Challenges Advocacy on key issues affecting young people’s sexual health rights often involves confronting deeply held attitudes and sociocultural and religious beliefs. This is a significant challenge but an essential component of achieving young people’s SRH rights – CSE, SRH services, access to abortion, etc. For several country programmes the challenge was to develop advocacy work on a national level, despite successes at a local level. Other challenges were to ensure relationships were in place with all of the key ministries: education, health and youth. The involvement of young people was present in some countries – notably Guatemala, the Philippines and Mozambique – but not developed in others. As noted in successes, involvement of young people can be critical to achieving results.
“It was very important to us to meet with parents and ensure the credibility of the centre.” Albania One of the challenges was inadequate human resources and the skills to under the systems and processes involved in governmental policy change. In Bangladesh, there is only one person in the advocacy unit at national headquarters and advocacy was not prioritised as part of the SALIN+ programme. While this was not necessarily a problem in the short term, in the longer term it will limit FPAB’s position and ability to achieve its goals. Engaging with parents and other gatekeepers in a range of communities (for example, parents in religious, poor, ethnic or rural communities) can be challenging. Sometimes programmes are reluctant to take on the major task of changing attitudes across diverse groups at local level. However, without visible support and champions, local programmes will have limited success. There is little evidence of the work achieved because data is not collected on this aspect to reflect the work achieved.
B Enabling environment 5 Policy and planning Summary Many of the countries in the SALIN+ programme are operating in a challenging national policy and social climate – in terms of SRH rights and access to services generally and often particularly in relation to young people. SALIN+ projects in the review had planning processes in place but more work is needed to ensure a rigorous planning process that involves meaningful input from young people.
5.1 Policy environment The policy context varied greatly from country to country, but each had its challenges. In Guatemala and the Philippines, for example, the Catholic and evangelical churches have considerable policy influence. In Bangladesh, Government service providers are not allowed to provide services to unmarried young people, and yet Government encourages NGOs to take the risk of provision. Abortion is illegal, but menstrual regulation is accepted practice. Both Mozambique and Tunisia had positive national government programmes of SRH service provision – in Tunisia for the whole population and in Mozambique a programme targeted for young
External evaluation of SALIN+ projects: Overview report 2010 25
people. Tunisia and Albania were the only countries with legalised abortion. Apart from some limited pilot initiatives to trial sex education (e.g. in Tirana schools) or established local initiatives (e.g. ‘Biz Generation’ in Mozambique), before the SALIN+ initiative, CSE was not on the national curriculum of any country. Even in countries such as Albania, with a relatively liberal policy climate in relation to SRH, the resources available for young people’s services are limited. As noted above (Advocacy 4.1), in the long term, without focused and successful advocacy programmes addressing SRH rights and CSE, IPPF country programmes will have a limited chance of making a significant difference.
5.2 Plans Achievements and successes Most countries had a planning process that takes into account the IPPF 5 ‘A’s. Some countries, such as Albania, have detailed SMART18 plans that consider young people’s issues under a number of headings. In Bangladesh, although the SALIN+ timeframe did not allow a thorough planning process, the lesson learned for their next programme, entitled ‘Unite for Body Rights’19 (UBR) and a thorough process is informing its development, including needs assessment, analysis of data and evidence of effectiveness as well as lessons learned from SALIN+ and the involvement of young people. The SALIN+ evaluation framework and report are being used in the Philippines to plan their Choices programme, as they have been in Albania. In Mozambique the Youth Action Movement (YAM20) developed and presented the SALIN+ programme proposal and took a strong leadership role in the development and delivery of the programme.
Challenges In a programme such as SALIN+ with limited time for initial planning, it is difficult to ensure rigorous processes that include needs assessment (including considering the needs of those who do not normally use Member Association services), provide evidence of effectiveness, meaningful engagement of young people and ensure all key partners have SMART targets. All countries in the review found ensuring a full planning process challenging in the time available for SALIN+ and many countries said that staff planning skills could be strengthened.
18 SMART: Specific, Measurable, Achievable, Realistic, Timed 19 An upcoming programme for young people in Bangladesh. 20 YAM is part of the global IPPF youth involvement programme
Several of the review countries included specific references and objectives relating to young people’s SRH, but often these appeared only in the adolescent section, as opposed to being integrated throughout the plans.
6 Information and evidence base Summary As identified above, carrying out needs assessment was challenging for all review countries in the short timeframe available for SALIN+. Many Member Associations found the additional data collection requirements for SALIN+ initially difficult and ultimately helpful. Data collection in many countries does not provide enough detail for effective targeting and organisational skills for analysis and use of data are lacking. More could be done across all review countries to make use of evidence of effectiveness and identify the most effective and cost effective components of programmes.
“SALIN+ – the reporting was double work, but it was useful because now we can measure the impact of the work we do.” Albania
6.1 Needs assessment Achievements and successes While the SALIN+ planning timeframe did not allow for in-depth needs assessment, many review countries learned the lesson of how important this process is for ensuring programmes meet local needs and have carried out careful and thorough needs assessment for their next programmes with young people (looking at sex, age, disadvantaged groups and other local factors). ACPD (Albania), FPAB (Bangladesh), FPOP (Philippines) and APROFAM (Guatemala) are good examples of where this lesson has been learned. During the review, ACPD and FPOP were both involved in needs assessment for their Choices programme, FPAB was for the Unite for Body Rights programme and AMODEFA in Mozambique for the A+ funds.
Challenges While all countries acknowledged the importance of needs assessment, the short timeframe for planning SALIN+ meant that this was not undertaken thoroughly. In particular, review countries did not tend to seek out and take into account the needs of those who do not currently make use of Member Association SRH
26 External evaluation of SALIN+ projects: Overview report
services which means there is a risk that some of the young people who most need access to services are missed out.
6.2 Data and information Achievements and successes Despite initially finding the additional data requirements of SALIN+ very challenging, all countries reported that once they had gained experience with the requirements and systems, that they found them to be a positive and helpful development that will allow them to assess more thoroughly the impact of their services and programmes. Throughout the period of the programme, the data and information collected were used in planning and monitoring of service delivery. In Albania, where previously the state relied only on their own data sources with nothing from the NGO field, the data becoming available through SALIN+ means that government has asked ACPD to take a national lead on coordinating data and information in some of this work. In Guatemala, staff and young people had collected a wealth of data from exit interviews with young people using services across the country, which was presented during the review. In Tunisia they carried out one survey with staff and users.
“After the data training – our stats were so good – higher – that people kept asking how we have achieved that. Now we’re being praised – but finally we are able to truly reflect the scope of our work and how hard we are working.” Mozambique Capacity building training (from IPPF Central and Regional Offices) on data collection, monitoring and analysis for staff was a positive development of the SALIN+ programme. In Mozambique, staff received training halfway through SALIN+, which impacted very positively on their ability to record and demonstrate the work they were doing – across the whole organisation, not just with the youth services. This training was not rolled out to other programmes.
Challenges The majority of the review countries acknowledged that their data collection and management information systems do not provide them with all the information they need to assess impact, point to underperforming services and identify unmet need. While it is difficult to consistently collect information on a range of factors
such as ethnicity, poverty/income levels and religion, it is important to collect age and sex disaggregated data as a matter of course. Sex disaggregated service data is not collected in all countries, which significantly limits the ability to understand whether the needs of both young men and women are being met. In addition, few countries collect and make use of detailed sex disaggregated human resource data on issues such as access to training by position, or recruitment and promotion by sex and age or other information to allow organisations to understand equalities issues within the workplace. In addition to the lack of data, the review found that staff generally do not always have the skills to make the best use of the data they do collect, or find it difficult to make use of data that appears to contradict their ‘common sense’ understanding of which initiatives are effective. For example, even more could be made of data collected in Guatemala’s exit interviews to see whether some services are underperforming or dig deeper into some findings (e.g. that cost of services does not matter to young people). In Bangladesh, to avoid discriminating against unmarried young people in service provision, marital status is not collected. This means that FPAB is challenged in its attempts to monitor and assess the success of their aim of providing services to unmarried young people.
6.3 Effective and innovative (see also 11, Accountability)
Achievements and successes Most countries reported seeking out some evidence of effective similar programmes during planning for SALIN+. All were eager to be innovative and identify promising practice in the SALIN+ programme. Systems were in place in all countries to monitor and learn lessons as the programme progressed. In Guatemala, team members recently had a field trip to Argentina to learn lessons from similar programmes there and the monitoring and evaluation team reported checking sites for evidence of promising or effective practice. In Bangladesh data and information were regularly sent to the young people’s team for analysis about what approaches were working and similarly in Albania, there are team meetings across all services to ensure approaches are working. AMODEFA in Mozambique was chosen as the ‘learning centre’ for all of the Lusophone21 countries and is developing work with all of the Member Associations in these countries to share their learning.
External evaluation of SALIN+ projects: Overview report 2010 27
Challenges It can be challenging to be rigorous about seeking out documented solutions to problems or evidence of best practice from regional partners, IPPF databases or more widely in evidence databases or internet searches. With probing, some interviewees mentioned that they reviewed evidence of effectiveness, but there appears to be no systematic approach to ensuring this is part of the planning process. Programmes generally did not have a systematic approach to contributing to the evidence base by carrying out in-depth evaluations or research. The review found that team members across the programme were not always clear about what constituted evidence of effectiveness.
7 Partnership and engagement Summary With some exceptions, across most review countries, partnerships with external core partners in health and education were productive and strong. There was generally less understanding of the need for wider partnerships and these were less well developed. While there were many good examples of engagement of key stakeholders and communities, this was not always systematic.
7.1 Core partnerships (see also communication) “APROFAM feels that the best way to work with young people is in alliance with other organisations – each doing its own specialised work but aligned.” Guatemala Achievements and successes To support the SRH and CSE outcomes of SALIN+, the core partnerships are state, NGO and private sector health and education organisations. Most country programmes had well developed partnerships across health and education that came together for advocacy and coordination of planning and service delivery. In Guatemala, APROFAM had a strong and effective partnership with the Ministry of Education, as well as with a range of partners for the successful advocacy programme for CSE. APROFAM also had a working relationship with Doctors without Borders, where they could send women for post-rape care.
In Bangladesh, partnerships were strongest at a local level, where state and NGO partners from health, education as well as a range of wider partners met monthly to ensure effective care pathways for young people. The Philippines also had very strong partnerships at a local level, with local government health departments and other community-based NGOs. Arrangements with local government departments provided FPOP with clinic premises, and links with health departments helped establish good referral pathways and joint work. They were less successful at establishing national partnerships. Albania had particularly strong partnerships with the Institute of Public Health at both national and local level, as well as with partners in health and education with whom they met for both advocacy and service planning purposes at national and local level. Albania has a partnerships section in its five-year plan, identifying how to strengthen them. It has up to date contact details, and has strong partnerships with 40 organisations and memorandums of understanding (MOUs) with 20 of these. Partners were quoted in the review as saying the coalition led by ACPD as being the only effective alliance in the country. Mozambique had a long-term working partnership with the crossministry ‘Biz Generation’ programme, developing work in schools, communities and clinical services. The partnership with the other NGOs involved was strengthened through the enhanced work of SALIN+.
Challenges Maintaining effective partnerships can be challenging in terms of the time and skills required, as well as avoiding competing for potentially scarce resources. Partnerships with government are also sensitive to governmental changes, including personnel changes. The Philippines had previously had good relationships with the Ministry of Health but with a change of government all of the staff they had established relationships with had been moved. In some cases, the Member Association had prioritised one core partner during the SALIN+ period. For example, in Guatemala, the Ministry of Health (MOH) does not have any focus on young people and is a relatively poorly resourced ministry (compared, for example, to education), although there are good clinical relationships between MOH and APROFAM. In the short term, prioritising education has been highly effective and this focus has likely contributed to the successful advocacy initiative on CSE. However, in the longer term, if the challenges of partnership with the Ministry of Health are not addressed, it could limit APROFAM’s ability to achieve its goals. Similarly in Bangladesh, FPAB did not
21 Lusophone countries include: Angola, Brazil, Cape Verde, East Timor, Guinea-Bissau, Mozambique, Portugal, Sao Tome and Principe & Macau (Portuguese speaking)
28 External evaluation of SALIN+ projects: Overview report
prioritise national partnerships for health and education during SALIN+; in the longer term this could limit the organisation’s potential to achieve goals.
7.2 Wider partnerships Achievements and successes There were several examples across the SALIN+ programme of effective wider partnerships that bring together organisations working on health and education for young people with others focused on wider factors that can have an impact on young people’s SRH choices, such as poverty reduction, drug misuse, violence against women. In Bangladesh, as noted above, local partnerships meet monthly to improve care and care pathways for young people. In Albania, for example, in the northern city of Shkodra, ACPD meets monthly in partnership with the local government social services department and a range of NGOs that deal with poverty to plan and delivery programmes and services. In the Philippines strong core partnerships were developed on a local level – with Local Government Units (LGUs), Chiefs of the Barangay (local government), Health Departments, other NGOs and community organisations. With the local partnerships the work of SALIN+ was able to develop more community-based work and become better established within local structures and local communities. Mozambique worked in close partnership with the other two main NGOs who are also part of delivering the ‘Biz Generation’ programme. Each had an identified work focus and met regularly to discuss progress, and most meetings included the Ministries and international partners.
Challenges While interviews showed that a number of the following are in place in some of the review countries, there is a dauntingly wide range of potential partnerships to explore systematically. These partnerships have the potential to allow targeted work with most at risk young people and also ensure young people are able to make positive choices. Some examples include: Domestic violence/sexual violence: shelters for survivors of
domestic violence, improved relationships with the police to ensure domestic/sexual violence is prosecuted Education: to ensure young parents can continue their education Specialist organisations that include support for young people – for example for commercial sex workers; lesbian, gay, bisexual
and transgender people; physical and learning disability; drug users, their partners and families Income generation: for young people without employment who are at risk of resorting to commercial sex work and drug sales/ use.
7.3 Civil society engagement (see also 4.2 and 4.3)
Achievements and successes There was some engagement with a range of stakeholders to support planning of services across the SALIN+ programme. For example, in Albania, ACPD’s wide range of partnerships helps the organisation engage with stakeholder communities.
“The work of AMODEFA is excellent – the peer educators are visibly excellent and so the ministry will continue to fund them because of their great work and the fantastic results.” Mozambique In Mozambique, AMODEFA’s involvement with the ‘Biz Generation’ programme has ensured strong links are in place with key service providers in health, education and the community (police, community leaders, youth services, etc.). This has supported the establishment of relationships with schools and remote communities.
Challenges While most review programmes had some engagement with communities, there tended to be relatively little engagement with representatives of those who do not generally use Member Association services (such as young people who are rural, working, illiterate and from risk groups such as sex workers, drug users etc.) and community leaders such as religious leaders, healthcare providers, local businesses, etc., as well as teachers and parents as outlined in section 4.2.
8 Capacity and resources (see also 3, institutional commitment)
External evaluation of SALIN+ projects: Overview report 2010 29
Summary
“Yes, I received training through SALIN. Now I feel confident working with young people.” Albania While all review countries included training and capacity building as part of the SALIN+ programme, some had better coverage of staff than others. Not all organisations had systematic arrangements to ensure service providers had the skills and attitudes to provide safe, effective youth-friendly services. While resources covered the programme during the SALIN+ period, sustainability of initiatives remains a significant concern.
8.1 Recruitment and job descriptions “A lesson we learned is that more training is needed – both for young people and adults. And without ‘clarifying values’, it doesn’t matter how many tarar melas and clinics there are.” Bangladesh Achievements and successes Member Associations were interested in the potential leverage that including a requirement to be youth-friendly in job descriptions. In Albania, this was already included in all job descriptions and is taken into account in recruitment.
Challenges The majority of Member Associations were not aware of the use of elements such as youth-friendly in job descriptions or in interview processes, and if they consider it, had employed it only in relation to youth workers and not the rest of Member Association staff.
8.2 Training Achievements and successes There were training programmes and values clarification components in all country programmes (see section 3.2). This exercise was easiest and most successful in terms of organisationwide change in the smaller Member Associations. For example, in Albania where there are only 19 staff in ACPD, all received training and values clarification across a range of issues relating to youthfriendly service provision.
“I work at National Office and I cannot believe how much I was affected by the VCAT training – I have my own struggle about abortion and it really helped me to think differently about the issues. It felt very important but then I go to church and the priest tells us we are abortionists and satanists … the course made me feel stronger. I’d like more.” Philippines In the Philippines with SALIN+, FPOP developed a strong training programme for peer educators, clinical staff, administrative staff and board members. Staff, both clinical and administrative, highlighted the impact and importance of this training.
Challenges In countries where the organisation has been operating for a long time with a large number of staff or where only those delivering the SALIN+ programme were trained, there was much less evidence of organisational change. This lack of training often went hand-in-hand with limited communication across the programme about the achievements of SALIN+, and resulted in limited organisational change. See also skills and supervision, below (8.3). Some organisations, such as FPAB in Bangladesh, neglected to train key gatekeepers such as cleaning, security and reception staff, but later recognised this gap and plan to address it in the future. Other review countries also did not recognise the need for and importance of this training.
8.3 Skills/Supervision Achievements and successes All review organisations had mechanisms for ensuring staff had skills to provide safe and effective services. Some organisations had systematic supervision of both paid and volunteer staff that assessed some or all among safety, effectiveness and youth-friendly service provision. In Bangladesh, for example, the system of monitoring employees includes whether staff have met targets; the comments and complaints procedure are taken into account as well as a ‘zero tolerance’ sexual harassment policy. Supervisors check the
30 External evaluation of SALIN+ projects: Overview report
percentage of young clients to see whether it is increasing in the SALIN+ project areas. In Mozambique as part of the ‘Biz Generation’ programme, AMODEFA linked into the support and supervision processes established within the wider programme. Following outreach events peer educators write up the event, identify any challenges and then do feedback to the coordinator.
Challenges Some of the Member Associations have a large range of services and centres but not all staff across Member Associations have the skills to deliver youth friendly services. Being youth-friendly is not generally a focus of supervision. For example, in Guatemala, there is evidence that some staff are not providing emergency contraception according to organisational or national guidance, but this is not being picked up by supervision. Staff in the non-youth focused clinics have not received training on meeting the SRH needs of young people and their rights.
8.4 Resources Achievements and successes All country respondents felt that the resources made available as part of the SALIN+ programme had resulted in a transformation in the participating organisations – in terms of some or all of the following: increased access to SRH services for young people, empowerment of young people and development of peer educators, a change in organisational culture, successful advocacy interventions and sexuality education and information.
Challenges Although there was considerable commitment across the SALIN+ programme to maintaining commitment to and services for young people, most Member Associations were concerned about the sustainability of the programme that had been put in place. Not all of the review countries had the resources to deliver youth-friendly services across the Member Association’s remit. For example, in Albania, there are no full or part time male staff, which means that there is no choice to see any male service provider. In Bangladesh the programme finished in December, resulting in cuts in numbers of peer educators and reductions in the salaries of youth counsellors. At the end of the review in Guatemala it was not clear whether the coupons for free services for young people would be able to continue. In Tunisia and Mozambique many staff were made unemployed at the end of the funding.
9 Communication See also 3, IEC
Summary While there are some examples of good communication with external partners, this could be more systematic. In most programmes, much more could have been done with the media to publicise services and successes and bring attention to advocacy issues. There was much room for improvement in internal communications across Member Association services to publicise successes, seek solutions for challenges and disseminate good practice. In all of the programmes there was a need for greater sharing of knowledge and lessons learned across the Member Association and the partners and there is a challenge there are processes in place to ensure this happens.
9.1 Partners – external Achievements and successes There are examples of good communication among stakeholders and other partners. Albania, for example, has memorandums of understanding with 20 of its 40 main partners that elaborate roles and responsibilities for ACPD and the partner organisation. These agreements were noted by partners as excellent in guiding and clarifying the relationship and enhancing service delivery. In Bangladesh local partnerships have monthly meetings in order to share experiences and plan services for young people. In Mozambique regular meetings are held with the three ministries and all of the other NGO partners of the ‘Biz Generation’ programme.
Challenges Some review organisations maintain relatively limited, unsystematic communication links with external partners and do not share experience and meet together to make plans for improved, joined up service provision. It can be a challenge and time-consuming to both develop and maintain the partnerships and external relationships but this needs to be prioritised in order to enjoy the benefits, such as joined-up service provision for clients.
9.2 Media Achievements and successes During SALIN+ there have been some good examples of work with the media to publicise successes, provide information about service access and SRH issues as well as bringing advocacy related issues to public attention for debate. In Guatemala, for example, APROFAM is sufficiently skilled in working with the media that they are able to turn negative publicity (such as Catholic Church attacks on the organisation for provision of SRH services) to their own advantage, using them as opportunities to publicise services. In Albania, there has been work involving young people on radio and television: six television
External evaluation of SALIN+ projects: Overview report 2010 31
programmes and two radio programmes were produced and released in 2010 on SRH issues for young people, made with the involvement of young people. There are also three Facebook sites, each linked to one of the city clinics, providing publicity and a means of engagement with young people. In Mozambique, as part of SALIN+ a memorandum of understanding was negotiated with the media that enables them to have a 50% reduction in publicity, and equally importantly helps to develop and maintain a strong relationship with various media people. The peer educators have developed radio work in the rural areas, where many people have no other media available and the peer educators have done educational sessions on the radio.
Challenges Most/all organisations in the review said that they had not made the best use of the media and were determined to do so in the future.
9.3 National network Achievements and successes – staff / internal Effective communication within organisations is critical in ensuring institutional commitment to provide consistently youth-friendly services. Successes with internal communication within Member Associations have been particularly strong in some of the smaller organisations. For example, Albania, with only 19 staff is able to meet regularly to discuss progress, solve problems and raise awareness. Many services relied on informal – and often irregular – communication and information sharing processes, as opposed to developing a systematic programme of internal and external organisational lesson sharing and learning.
Challenges In many organisations there is inconsistency of communication between staff providing specialist young people’s services and generalist staff within the organisation. In Guatemala, for example, despite the youth and general clinics being next door to each other, information on good practice on how to ensure YF services is not shared effectively between the two. In Bangladesh, there appears to be relatively limited communication about the SALIN+ programmes and progress during the initiative across the whole of FPAB, both to share best practice and to ensure all FPAB staff develop and understanding of and commitment to YF services and young people’s SRH rights and access.
“We had to overcome a lot of challenges to create awareness to be allowed to give services to unmarried young people. Young people are now not so vulnerable. The whole idea of providing services to young people is quite unusual in Bangladesh.” Bangladesh
10 Service delivery Summary In all of the centres in the review, a wide range of services was available, with some inconsistencies and some lack of availability or governmental permission to offer specific services. On the whole, the inconsistencies were linked to staff having personal beliefs that opposed the use of emergency contraception or the provision of pre or post abortion support. While most clinics said they would never turn away anyone who asked for services at their centre, for some SALIN+ programmes there was a focus had primarily focused on young people in schools and universities, so with some exceptions, many IPPF priority were not prioritised for access. Some programmes focused more on the provision of education and information and less clinical provision. All of the SALIN+ programmes had a clear understanding of the basics of providing an accessible and youth-friendly service – including location, publicity, good opening hours, confidentiality, responsive services and non-judgmental staff. The majority aimed to offer this but there were also inconsistencies with some aspects – staff attitudes, confidentiality, etc.
32 External evaluation of SALIN+ projects: Overview report
10.1 Range – of services “The entertainers have a great need for access to abortions but it is illegal. The closest countries to get one are Thailand and Hong Kong but many women will use illegal methods – often with disastrous results. They know of one clinic where abortion is available and refer each other there. It really should be available with FPOP but there is no support for any doctor to stick his/her neck out and do it.” Philippines Achievements and successes For most of the Member Associations, there was a balance of clinical, CSE and IEC service provision. Clinics offered core services, including contraception, condoms, emergency contraception (EC) (or replacements for EC), pregnancy testing, pre and post test counselling and support; maternal care; treatment for other gynaecological problems; cervical smear tests; Pap tests; services for young people who have experienced sexual, physical or emotional violence; pre and post abortion care; voluntary counselling for young people and limited treatment for male SRH concerns. In many review countries, peer educators are involved in general counselling. None of the Associations are known to offer abortion, including Albania and Tunisia where abortion is legal but is provided only by the state family planning services (and in the case of Albania, private and other). ATSR and ACPD refer young women to the state services. In Albania and Tunisia abortion is legal. In Bangladesh menstrual regulation is legal, but abortion is not.
Challenges In Guatemala, Mozambique and the Philippines abortion is not currently legal but there is a recognition of the importance of advocating for legalisation, especially with high levels of maternal mortality, a challenging step to take in most countries. All of the youth-friendly clinics should offer the full range of services available (in that country) and should be fully supported by their governing boards to ensure this is possible. However, in several countries there are cultural or religious beliefs that challenge the provision of a full service and ensuring a consistent staff
approach in relation to young people’s SRH rights and access to youth-friendly services.
10.2 Range – of clientele Achievements and successes Although – apart from young people in schools and universities – there was limited specific targeting or identification of key groups of underserved young people, there were examples of the services being accessed by a relatively wide range of clients in most programmes. Considerable efforts have been made to reach both male and female young people through outreach and promotional activities. In Mozambique SALIN+ enabled AMODEFA for the first time to offer services to young people in very remote areas and also to develop strong work with young deaf people. In Tunisia, work with young people who are deaf and also those who have mental ill health was central to the work in Medenine. In Tunisia centres were well equipped with computers and Internet access as a means of attracting young people into them. In the Philippines services were located near to where the sex workers worked and in the market where many young people worked. In both Bangladesh and Albania, strong and effective partnerships with a range of community and non-governmental organisations meant that many more priority groups were reached for SRH services than the Member Associations would have had the capacity to reach otherwise. In Guatemala, partnership with NGOs serving homeless young people and young people from rural areas increased access to target groups.
Challenges Without in-depth needs assessment or data to know exactly who is using the services, it is challenging to plan and deliver services to the young people most marginalised and underserved. Most of the programmes struggled to get young men into the clinical services, even if there was a gender balance with the peer educators. The use of services by young people who are working/ rural/ illiterate/ gay, lesbian, bisexual or transgender, and other marginalised young people is limited and not a focus of most programmes.
10.3 Accessible, youth friendly services Achievements and successes The key elements of assuring accessible, confidential, nonjudgemental staff attitudes and responsive services have been central in all of the programmes. There was a high level of staff understanding of the need for these core elements of accessible, youth-friendly services, which offer. Offering a wide range of services, having good locations and youth-appropriate opening
External evaluation of SALIN+ projects: Overview report 2010 33
hours were all considered in developing the services. Most centres had identified clear pathways for referral of the young people to other required services. Some centres were restricted in what they could offer because of small (and in some cases, minute) clinics and the difficulties of ensuring confidentiality in a one-room centre. In many centres the IPPF SRH rights poster was visible, highlighting to young people their rights to a confidential and sympathetic service. In Guatemala APROFAM developed a system of coupons, which were distributed during outreach work and enabled young people to access totally free services, was seen as a key reason for more young people using the clinics. Other centres offered free or reduced costs to young people without coupons but all of organisations were concerned that without dedicated funding to support free services, the numbers of young people would be greatly reduced. In Bangladesh, the tarar melas (youth centres) – generally attached to the clinics – provided ‘cover’ for young people, by providing a range of services such as computer and English language training, competitions, music and dancing. This meant that young people were able to enter the clinic without community members knowing why they were there. Similarly, in Albania, the young people’s centres (with resource centres, discussion groups, book clubs – reading books with SRH themes) served as what one young man called “camouflage” for young people using services. There are many examples of young people being involved in monitoring services, including exit interviews in Guatemala, Albania and Bangladesh, and youth researchers and young members of the Project Monitoring Committee in Bangladesh, and similar bodies in Mozambique, the Philippines and Guatemala.
Challenges A major challenge is to ensure consistency of services and staff approaches and attitudes across all Member Association services, so that all provision is youth-friendly, not just clinics identified as specific youth centres. Another challenge is to ensure all staff take very seriously the need for services to young people to be as confidential as possible. In addition there is a need to improve reception confidentiality, to promote confidentiality more proactively and ensure all staff fully understand the policy and use of it. Clinics have a challenge to provide real confidentiality, at times due to structural limitations – low ceilings, no doors between rooms (just curtains) and very small spaces. This reflects a very difficult problem of how to gain and give information in an open space when young people may feel shy and/or embarrassed.
11 Accountability Summary Monitoring was generally good although some additional data collection and involvement of young people could further improve progress. There is relatively little in-depth evaluation. Many countries learned many useful lessons from the programme, but without a systematic approach to lesson learning. Governance is very mixed, with some boards involving young people in a meaningful way and others not yet having any young members, despite IPPF’s requirement that 20% of board members are under 25.
11.1 Monitoring and Evaluation See also 6, information and evidence base.
Achievements and successes Monitoring is required of all participants of the SALIN+ programme. As noted above, there were additional requirements beyond those usually undertaken by Member Associations and ultimately most review countries were pleased with these conditions. In terms of data about quality, some programmes instituted a range of measures to ensure that the views of young people were collected, many of which were well-recognised data collection mechanisms, including exit interviews, suggestion boxes and complaints procedures. Guatemala carried out an extensive exit interview and presented findings to the review team. Bangladesh had male and female youth researchers and youth involvement on Project Monitoring Committee. Many of the review countries had trained young people to be involved in monitoring and made a point of capturing young people’s views of services (see also 10.3).
Challenges As noted above in information and (element 6) evidence base, further data collection would strengthen organisations ability to plan, deliver and monitor services. Time, resource and staff skill limitations make it difficult for centres to carry out regular exit interviews and then analyse and use the data to improve services. No Member Associations in the review had systematic approaches to more in-depth evaluation or to identifying cost effectiveness or success of interventions.
11.2 Lesson learning Achievements and successes While many of the Member Associations noted that there was not sufficient time to plan the SALIN+ programme, most/all had learned useful lessons that they intend to use in current or future
34 External evaluation of SALIN+ projects: Overview report
planning of young people’s services. This includes Bangladesh in its plans for Unite for Body Rights, an upcoming programme for young people, and Albania and the Philippines’ plans for Choices. Lessons noted by SALIN+ country staff include more involvement of young people in planning and monitoring, doing needs assessment in all programme planning and involving gatekeeper staff in training and values clarification.
Challenges As noted in 9 – Communication, many of the Member Associations did not communicate effectively about the successes and challenges of SALIN+, particularly in large organisations where the SALIN+ programme was not rolled out across all the organisation’s services. In addition, there was limited evidence of organisations sharing information about SALIN+ with partners and publishing findings and progress in journals or the Internet. AMODEFA are putting together a ‘book of good practice’ to promote and share with other organisations the achievements they have had in their long-term work with young people.
11.3 Governance Do members listen to the young people on the Board? “Yes! Beginnings are hard. I was inexperienced and found it a bit difficult to express my views freely. But after several meetings the other Board members welcomed my thoughts. It is an achievement for both the young Board members. We make recommendations and express our views.” Albania Achievements and successes Most countries had a gender balance on their boards and some had achieved the 20% board membership under-25 years of age. There were some good examples of board members that had meaningfully involved young people. For example, in Albania, youth representatives interviewed as part of the review. Procedures are generally in place for staff and/or young people to take issues to the board.
Challenges Some countries in the review had not yet recruited youth members to their boards. Where there is not yet engagement with this
agenda there may be a risk that any young people taken onto the board will not be able to have meaningful input. There is evidence that in some limited cases, board members do not agree with the IPPF principle of young people’s SRH rights or with all of the IPPF Five ‘A’s. In Bangladesh, some board members expressed the view that unmarried young people should not receive services. In the Philippines there was resistance from some Board members to work fully with the Five ‘A’s and fully engage with all of the issues in relation to young people’s SRH rights.
C Crosscutting elements
IPPF introduced three key issues to the review framework – sustainability, promising practice and added value. These were not rated, but were considered throughout the review and, not surprisingly, sustainability was brought up by a number of interviewees.
Sustainability Sustainability is a critical aspect if youth-friendly services are to be delivered consistently over time. However, the overall global SALIN+ programme does not appear to have been developed with sustainability as a primary focus. While the nature of the programme – one-off funding for a limited period of time – does not necessarily encourage sustainability, it is possible to take an approach that strongly encourages this. Most countries have tried to incorporate approaches from the SALIN+ programme into their annual workplan for 2011, although considerable reductions will be inevitable even in countries with follow-on funding (e.g. Choices and ‘A+’). Organisations will face significant challenges to continue to deliver high quality, free, youth-friendly services without extra funds. Many of the review countries have developed programmes that rely on relatively substantial, additional funds to be able to provide free services to young people. Measures that have been central to SALIN+ programming such as coupons for free access (Guatemala), free services (Bangladesh, Albania, Mozambique), computers, internet access and computer games (Bangladesh, Guatemala, Tunisia), multiple new centres across the country (Bangladesh, Tunisia, Mozambique), will all be difficult to maintain without ongoing increased levels of funding, which, at point of writing, is not going to be available in subsequent young people’s programmes such as Choices or A+. In many countries, there are already reductions (e.g. in Bangladesh and Tunisia, the computers remain, but there are no funds for internet access; youth counsellors have had their salaries halved; in both Mozambique and Tunisia much of the skilled youth staff have been lost).
External evaluation of SALIN+ projects: Overview report 2010 35
Cost effectiveness of interventions has not been a focus of planning, research or review. In fact, only one interviewee mentioned cost effectiveness. Member Association level fundraising is done by most/all organisations to an extent, but there is less evidence of this at a local service provider or city level. Use of mechanisms such as sliding scale fees was viewed negatively by some organisations. Some organisations expressed the view that it is important to charge for services, both to ensure people value the service, but also to generate income to support those who cannot pay. For some of the Member Associations the current move from core funding to programme and one-off money presents considerable challenges when the skills to secure funds are not present. Sustainability is not all about money, which is why this review considered a range of basic institutional systems and attitudes that can help contribute to sustainable delivery of youth-friendly services.
Promising practice A range of promising practice was suggested based on observation and some monitoring. However, because detailed evaluation was not undertaken from the beginning of SALIN+, it will be important in future to ensure a more systematic approach to identifying and understanding the critical elements to successful programmes for young people. Peer educators were seen as the most effective way of reaching underserved young people, although most/all projects acknowledged that the most underserved young people were not reached by SALIN+. In all/most programmes, the peer education programmes were noted as having empowered young people and attracted them to services, as well as adding value to initiatives ranging from advocacy to outreach to Facebook use to attract young people’s interest. Promising practice identified in each country included the following:
Albania Youth centres and peer educators were seen by all interviewees as key components to the success of the SALIN+ programme in Albania, allowing young people to develop skills and understanding of SRH and wider issues, as well as providing “camouflage” to young people to access services.
developing centres to attract young people and provide
‘camouflage’ for young people using clinic services partnerships contracts to outline roles and responsibilities partnerships for effective coordination of service provision partnerships that consider wider factors than SRH – including
poverty, literacy, trafficking, drug use, etc. building on partnerships already in place – e.g. with Roma. have memorandums of understanding (MOU) in place for
partnerships
Bangladesh In Bangladesh, the response was exceptionally positive across virtually every interviewee (young people, parents, teachers, government and NGO partners, staff, board members) to the tarar mela’s (youth centre) integrated youth centre (capacity building, access to information and services22 and socialising) approach. There was also widespread agreement that the youth counsellors were an effective way to reach large numbers of young people with counselling and emotional support. As part of the review, the Youth Researchers identified the following components of promising practice: young people giving services to young people counsellors very friendly welcomed, not viewed negatively confidentiality is maintained games[are provided]/free [service] non-judgmental attitude can have fun – no adults around don’t collect information that is not required – e.g. parents’
names (FPAB found that they had been asking too much information. Even through there was no space on the form for parents’ name, staff were still asking for it). Now they are using serial number, name and service.
Guatemala Interviewees throughout the review noted that the use of coupons by peer educators to attract young people into the clinics with the offer of free services, was critical to the success of the programme. Workshop comments on promising practice included health fairs (e.g. 1,000 students attending) where APROFAM
Some promising practice mentioned by interviewees included: use of peer educators
give information, especially on the special days (e.g. International Women’s Day). They can trace the place from which the coupons are given (fairs, outreach work, etc.). The
22 The only point of contention raised by an FPAB board member and some teachers interviewed was that unmarried young people should not have access to services. They voiced the view that they spoke for parents who would not accept young people having access to services. This did not accord, however, with findings from parents.
36 External evaluation of SALIN+ projects: Overview report
peer educators could put their names on the coupons, as well as the place. The clinics do ask how a new young client found out about the clinic; the young client usually gives the name of the person who referred him/her. promoting services at the educational talks working in a network with groups of youth peer education: In relation to underserved young people, the peer educators are the greatest link.
Mozambique Mozambique had very strong foundations and work in place before SALIN+ and took the opportunity to strengthen and enhance this work, particularly with the work of the peer educators, young deaf people and young people in remote, rural communities. Interview and workshop comments on promising practice included:
links with local government and partnership with health centres
and staff.
Tunisia Staff and volunteers noted positive achievements despite the new services only operating for a short period and the impact of ‘The Revolution’. 23 Interview and workshop comments on promising practice included: creativity – sexual health monopoly, work with deaf young
people, mental health issues preserve the relationships and exchanges between organisations promote activities more – dance, theatre, drama, etc. do more research on needs involvement of adult volunteers – working in partnership with
young people need to build on the lessons of the youth-friendly services,
particularly the work with the mobile van and outreach work to bring in more young people and also support engaging more peer educators. the quality of the youth services should impact on all of the other services AMODEFA provides. the level of participation of young people and also of the Board the work with hearing impaired young people, in remote areas and with vulnerable groups the ability to expand the work into other provinces the involvement of young people, allowing good communication and the training of young people to provide services the quality of the youth-friendly services provided.
Philippines Staff, volunteers and partners were all positive and enthusiastic about the achievements of SALIN+ and the lessons they had learned from the work. As most of the work was new to FPOP, it had a major and positive impact on the organisation as a whole. Interview and workshop comments on promising practice included development of the youth volunteers and training to wider FPOP
staff team the link of FPOP services to the colleges of nursing – both
organisations benefit and there is a larger core of trained nurses targeted outreach with young entertainers, working in bars and
clubs and offering their ‘work registration’ check-ups working in a network with groups of youth organisations (e.g. Foundation for Adolescent Development [FAD])
increased peer education at all levels of the project local partnerships more people involved and services more known ability to respond to the needs of young people.
Added value The review sought to identify added value. Many of the programmes had positive unexpected results, some of which have greatly strengthened the work of the Member Association. Other added value looked at was the impact of the SALIN+ work on the wider national health systems and any provision of youth-friendly public sector service provision.
Albania SALIN+ has made ACPD realise the importance of providing
services to young men. Increased data requirements have meant that ACPD will now be
better able to measure the impact of their work. ACPD has realised the importance of outreach to ensure it
meets the needs of underserved groups. ACPD’s advocacy for CSE has brought the organisation further
to the attention of the government and has developed and improved its reputation.
Bangladesh In Faridpur (and possibly elsewhere) the project encourages
birth registration, which is an important factor in reducing early marriage. The district officer of Faridpur felt that the infant mortality rate had fallen.
23 In December 2010 to January 2011 Tunisia experienced civil unrest that resulted in the overthrow of the previous regime and an interim government until a democratic government is elected. This period is referred to as ‘The Revolution’.
External evaluation of SALIN+ projects: Overview report 2010 37
‘Eve teasing’ (harassment of women) has gone down. The
training at the tarar mela (youth centres) has had a great impact in raising awareness. Young people’s confidence has increased. One young person mentioned being able to ask her father to buy sanitary napkins for her (an accomplishment in any country!). There was a widespread view that the aspiration of young people using the tarar melas (youth centres) – and particularly those who were trained as peer educators, youth counsellors, youth researchers – had increased.
Guatemala The new law on sex education was passed by the Congress. Young people go with parents so they find out about the
services. “We didn’t expect so much support from the media. They
always come and information is presented in the radio, TV, etc.” Young people are rising up, they are getting more presence.
One of the staff has good contacts with the media.”
Mozambique Increased outreach work to vulnerable groups and the visibility
of the work the training of professionals to work with young people the overall image of AMODEFA with the radio programmes –
they need to continue the positive relationships established and the increased numbers
using the services the real strengthening of the young people’s programme the efforts of all of the AMODEFA staff in promoting the values
of AMODEFA and the need to be committed the positive relationship with the government and increased
links to state services, including hospital-based youth-friendly services.
Philippines the raised profile of FPOP across the Philippines as a provider of
services to young people the young entertainers become regular service users following
their ‘work registration medical’ check-up local partnerships with local government, health departments
and NGOs – including the local government supporting local work with money for events, provision of premises and support to further develop youth-friendly services the importance of work achieved in advocating for the Reproductive Health Bill and the national knowledge of the role of the peer educators.
Tunisia positive relationships and increased numbers of young people
using the services the image of the institution with more publicity and the need to
continue this the positive relationship with the ONFP and other stakeholders.
38 External evaluation of SALIN+ projects: Overview report
5 Conclusions, lessons and recommendations 5.1 Conclusions and lessons learned SALIN+ offered IPPF and the Member Associations a great opportunity to strengthen and increase their work with and for young people. All of the programmes have done an impressive job in a very short time of developing and implementing a wide range of youth services. At the same time, to have sudden large amounts of money can often be a ‘poisoned chalice’. Significant amounts of one-off funding with no assurance of continued resources can be a difficult challenge for any organisation tasked with delivering. Despite this, all of the programmes reviewed felt that SALIN+ had been ‘worth it’, though some added a ‘but’. The ‘buts’ were connected with the lack of adequate planning time, the lack of relevant systems in some organisations, the challenges with communication and the issue of sustainability. The organisations that already had some of the systems in place – planning, data collection, partnerships, communication, etc. – faced fewer challenges. For the majority of the SALIN+ Member Associations, the experience was part of the global transition from unrestricted to restricted funding. For many this was a first experience of focusing on outcomes and relatively extensive data collection. The majority of Member Associations were not accustomed to working to IPPF centrally-set objectives, funding time restrictions with no carry-forward available or substantial data-set and format; some organisations struggled with this. This is a major strategic change and organisations would have benefitted from an increased skillsset – planning, data analysis, fundraising, effective partnership development, etc (as outlined in the framework). SALIN+ supported the majority of Member Associations in this global transition to more restricted funding programmes that is taking place both inside and outside of IPPF. But at all levels there is a need to identify ways of supporting the Member Associations in this transition. It is not entirely accurate to do direct comparisons among the SALIN+ programmes and the ratings they have received because they represent such distinct demographics and local priorities and needs. However, the review and the ratings show broadly the differences between organisations. For example, where one organisation has received a 3 for governance and another a 1+, the former would have been found to have systematically in place many more of the components of that element. There were some strong areas of achievement across the programme, including the enhancement of clinical and outreach services, augmenting of peer education programmes, and increased advocacy and partnership working. Some of the areas of lesson learning – some identified by recognising gaps – are discussed below.
Planning All of the Member Associations found the SALIN+ planning challenging, particularly in context of the short timeframes but also in terms of the skills to carry out needs assessments, services reviews and the analysis of these. However they all learned and are better prepared for planning and developing future work. In addition, there appeared to be minimal linkage with either other restricted projects or core work.
Information and evidence base SALIN+ gave the opportunity to develop and implement more extensive systems for collecting data and some of the centres learned a lot from this. Extensive needs assessment did not happen (mainly due to time constraints) but there was a solid understanding of the need for this, although possibly a lack of skills to effectively carry it out. A key example is Mozambique, where following capacity building training, FPOP not only improved the SALIN+ data reporting but the reporting for the whole of the organisation. All of the SALIN+ programmes might have benefitted from this training. Gaps for all centres included the collection of information on who was using the services – sex, ethnicity or other specific groups, poverty, etc. Cost-effectiveness was not considered but for many programmes there was much discussion on how to offer the services free or as cheaply as possible; research or analysis of these issues was not present.
Partnership and engagement There were some very positive examples of partnership working and the majority of these (e.g. Mozambique, Guatemala, Albania) were built on partnerships already in place before SALIN+. All of these existing partnerships were strengthened through the SALIN+ work and partners noted this. But even where partnerships were in place – and with some exceptions, such as Albania – they often were not multi-dimensional but focused on one aspect (e.g. SRH), seldom reflecting community-based partnerships (income generation, social services, HIV, shelters). Engagement with parents and other gatekeepers was not developed as fully as would have been beneficial, but where when it did happen it had a very positive impact on the programme work: involving parents and religious leaders in breaking down taboos and advocating for young people’s SRH rights and supporting their daughters and sons. Though there were usually referral pathways in place and links to other health services, they were seldom focused on specific needs of young people (e.g. police in relation to sexual assault, education in relation to the rights of young pregnant women, agencies for domestic violence, income generation schemes to address poverty).
External evaluation of SALIN+ projects: Overview report 2010 39
Communication In some countries there were clear systems in place to support the work with external partners and to jointly share lessons. Several programmes realised the benefits of using the media to support and promote their work to the wider community and the services to young people. Communication between SALIN+ programmes, between country partners, between Regional and Central Office – etc – need strengthening to make them more effective. The reviewers shared experiences from other countries when carrying out reviews but realised that this was not happening on any planned level. Both staff and young people appreciated this information sharing and learning of SALIN+ work in other countries. There were some frustrations with perceived mixed messages of information received from and within the IPPF Central Office and Regional Offices particularly in relation to finances, where country staff at the operational level were not clear about the rationale for financial decisions (such as why some funding ended in December as opposed to March). There also appeared to have been cases where communication between finance and project staff at central level could have been improved At the same time, frustrations exist at central and regional levels that repeated messages do not appear to be well absorbed. In countries where English is not the common language, there were additional frustrations and barriers in receiving information all in English. There was also often poor communication within organisations to clarify guidance and principles, share lessons and best practice, publicise the successes of SALIN+. For example, several country programmes noted that writing up successes was an additional burden (while acknowledging that this does need to be done). This lack of communication will have repercussions beyond the impact felt on SALIN+ and unless improved, will continue to impede success.
Organisational capacity, resources and development As noted above, the Member Associations that had the systems and ‘basics’ in place, were able to deal better with the challenges of the SALIN+ programme. Others needed capacity building in relation to the skills of planning, data analysis, effective service costing, ensuring youth-friendly staff (human resources), etc. In many countries, there are few examples of using job descriptions and recruitment processes to ensure staff and volunteers are youthfriendly and pro-choice. Training was an essential element of not only the peer education volunteers but also as means of ensuring all staff – clinical, administrative, support – understand and accept young people’s
SRH rights and needs. Some of the training was ad hoc and not within a recognised quality training framework or followed up with performance management or quality assurance measures to ensure learning had occurred.
Sustainability and coverage In every country the delivery of youth-friendly services was increased and diversified and there were positive examples of new groups of young people using the services, especially through outreach work. In some countries with cultural and religious barriers, there were inconsistencies in context of universal access for young people to all services (e.g. doctors unwilling to offer emergency contraception or offer pre or post-abortion care due to personal views). A one-off programme such as SALIN+ is likely to develop a number of high profile, relatively high cost, specialist centres, such as the youth centres developed in many countries. While ensuring minimum standards across all services is unlikely to be as newsworthy or appealing to young people, these centres do not ensure the widest access possible to youth-friendly services. Sustainability was the biggest challenge for all of the reviewed SALIN+ programme Member Associations – how to continue the work (particularly the high profile youth centres) but with much less funding, and usually reduced staffing levels. Several of the programmes provided free services (some with coupons, other free to all attending the clinics) and much discussion centred on the impact the loss of this would have on young people’s ability to pay for and use the services or whether there is a need for ‘sliding scales’ with some assessment of ability to pay. There was an increasing recognition of the need for fundraising but also of a need for the skills and knowledge to do this effectively. This relates both to core and project funding, but organisations will need to cover increasingly greater proportions of core funding.
Accountability and governance Most countries have achieved the IPPF standard that 20% of the governing board members should be under 25. There are issues in some cases about how meaningful their participation is, and concerns that it could be a token presence without real participation. Gender balance was not always achieved in boards. There were examples where individual members of the governing board were clearly not supportive of young people SRH rights, nor of the provision of services for all young people.
Member Associations For summary conclusions relating to individual country programmes, see appendix 3. For full details, see country report.
40 External evaluation of SALIN+ projects: Overview report
Regional Office The review did not include specific work with the Regional Offices, although the youth focal lead from each Region was interviewed at the beginning of the process. The involvement of Regional Office in the SALIN+ programmes ranged from supportive and hands-on to just being the route for reporting.
IPPF Central Office Central Office of IPPF experienced some of the same issues that were identified in the Member Associations, such as time and capacity limitations for planning, reviewing and overseeing. When skills and training were provided, the outcomes were positive but this was limited. Communication and dissemination of news and information often did not reach the programmes. For example, IPPF has clear guidelines on use of language in IEC materials but few of the SALIN+ programmes visited had seen the guidelines or they were not available in the relevant languages.
5.2 Recommendations Overarching recommendations Maintain momentum The first recommendation that has been made in all country reports and is repeated here is to keep up – and build on – the good work. As noted above, there are many positive developments that can point the way to improved SRH rights and access to SRH services and comprehensive sexuality education. However, to be able to do this – with reducing resources – it will important to be rigorous about identifying what the key factors for successes are and to ensure the basics are in place across Central Office, Regions, Member Associations and local service providers that will allow IPPF to achieve its goals.
Member Associations For summary recommendations for member associations, see appendix 3 and for details, see country report.
Regions Regional Offices, with their central coordination and performance management role, should consider how the following recommendations made for IPPF Central Office impact on them and how, working with Central Office and Member Associations, these recommendations can best be achieved. Of particular importance is the role regional Offices take in relation ensuring effective flows of information and communication about programme and contract issues, as well as performance
management. An example is the internal communication role between youth focal points and finance to ensure all those concerned are aware of the terms and conditions of any programme and respond appropriately. Regions are well placed to increase a focus on evidence-based practice and quality.
IPPF Central Office The following crosscutting recommendations emerge from the review for the Central Office:
Planning While there was very limited time for planning the SALIN+ programme, many Member Associations have learned lessons about how to approach planning, ensuring that organisations are ready to respond. At Central Office level, this may include building into planning elements such as midterm and end of programme review better opportunities for lesson learning and communication
across the programme further data requirements (see below) capacity building for core functions (e.g. as outlined in the
enabling environment component of this review framework). It will be important that this begins before funding is received to ensure capability to deliver effectively. ensuring planning takes into account other programming across IPPF, both restricted and core.
Information and evidence base As noted above, this transition period is an opportunity to deal with the move towards more restricted funding and outcomesfocused work. It will be important to measure and demonstrate the effectiveness and cost effectiveness of interventions. The review found that – despite initial resistance to increased data requirements – a number of Member Associations expressed the view that these measures would help organisations to better measure the impact of their work. Data and information: it will be important to collect
information that will allow IPPF to determine whether it is reaching its priority groups – and both young men and young women within these. This includes items such as sex disaggregated data for a range of service delivery on an ongoing basis; means of occasionally assessing coverage of harder to measure factors, such as poverty, ethnicity, etc. In human resources it will be useful to collect sex disaggregated data on areas such as recruitment, promotion, placement and training to ensure that clients get access to a choice of male and female providers.
External evaluation of SALIN+ projects: Overview report 2010 41
Evidence and innovation: In order to address the lack of focus
regular, timely, recorded, formal communication about contract
on the making use of available evidence of effectiveness to inform planning and interventions and to ensure organisations contribute to building a meaningful evidence base, IPPF should find ways to make sharing lessons about successes and dealing with challenges, identifying effective and cost effective practice and supporting organisations to find cost effective ways of doing more in depth research and evaluation (such as working with local and international universities to investigate the effectiveness of interventions). This emphasis on evidence of effectiveness – combined with effective supervision and performance management – can contribute to an increased focus on quality.
arrangements, funding and finance guidelines, with follow-up dates and deadlines regular written, electronic programme updates for partners and meetings for sharing practice and lessons light touch, informal, interactive communication processes for updating, problem solving (e.g. through email and Skype networks).)
Partnership and engagement To ensure IPPF is not a ‘silo’ programme, IPPF should support their Member Associations with examples of effective and low cost (in terms of resources, staff time etc.) partnership working that contributes to planning, service delivery (for example care pathways for service users, joined up provision), monitoring and evaluation and advocacy. Basic measures include up-to-date contact details, regular communication and meetings with partners with standard agendas and minutes with actions that are followed up. Stronger organisations have a partnership plan or include an approach to partnership in their strategy. It is important that these examples cover both core partnerships (i.e. health and education) and wider partnerships that can impact on whether young people are able to make healthy life decisions (e.g. organisations dealing with income generation, housing, drug use, HIV/AIDS, sex work, universities, youth organisations – and many more).
While IPPF guidelines for language and terminology are very useful (though as noted, not widely known), IPPF should ensure that the following are also in place: guidelines for creating youth-friendly publications (i.e. youth-
friendly YF text/illustration balance; illustrations that reflect diverse young people’s experience, language that is acceptable to/understandable by young people; taking into account high illiteracy levels in many countries served by IPPF) guidelines that take into account issues such as zz the low bandwidth available into many countries, so that documents are not slow or impossible to download or use zz the importance of producing materials that can be printed or photocopied in black and white zz ensuring countries where English is not the first – or common – language can access resources (i.e. translation at same time of English editions)..
Sustainability and coverage As noted above, sustainability must not just be a word – it must be meaningful. IPPF should ensure that organisations have the capacity and creativity to: raise funds at national and local level (e.g. through seeking
IPPF should build community engagement more strongly into programme requirements to ensure local level acceptability of initiatives. It can also be useful occasionally to monitor the effectiveness of partnerships, either formally or informally.
Communication Improvements in communication are required at all levels. It is essential to ensure an effective two – way flow of information from central to regional to country to local level and back. While, as noted above, communication can often be challenging, it is important to develop creative, useable, non-burdensome approaches to communication that are appropriate to the message, for example:
funds from local businesses, government and affluent citizens for multidisciplinary centres such as the tarar melas [youth centres] with the theme ‘supporting the future of X city’) understand value for money and cost effectiveness and how to measure them how to cost services and interventions explore costing regimes to ensure as much as possible that costs are covered and those with more money are paying for the services of those with less capacity building down to local level (e.g. local champions/ funders etc.) to ensure fundraising and related skills are well embedded in country organisations linkages between other projects and core funded projects should be in place from the start As noted above, this type of short term, one-off funding is likely to result in relatively unsustainable multidisciplinary centres that are very well-regarded by all review interviewees, but reach limited
42 External evaluation of SALIN+ projects: Overview report
Exclusive young people clinics/ services
SRH services with some young people’s provision
Government/ private general health clinics/ pharmacies/ NGOs/outreach
Member Association generic SRH services
low coverage/maximum YF
high coverage/minimum YF
Figure 1: Continuum approach to youth-friendly services developed by the review team
numbers of clients and are difficult to sustain. In addition, although a focus on young people in schools and universities is entirely reasonable in the timeframe of this project, all programmes will need to be much better at targeting in the future. This review has recommended the continuum model in all countries visited. For details, see below. Project funding can be much more complex and difficult to manage and deliver than core funding. In addition, project funding is new to some of the Member Associations. As noted above, it will be important to have standard components in place when planning programmes with such funding. It will be critical to ensure that sustainability is built meaningfully into project delivery to avoid raising expectation that is not achievable in the longer term. With reduced resourcing, it is important to be rigorous about identifying the key priorities for ongoing work and ensure that all of the services in every Member Association are at least essentially youthfriendly. This should not be dependent on special funds.
To ensure organisations provide youth-friendly services it may be helpful to identify a quality training framework and programme to use across all Member Associations. To support ongoing learning, it is possible to develop a set of integrated training cards that include a wide range of methods (role play, quizzes, checklists, case studies, planning) that can be discussed and employed in 20 minutes during regular staff meetings. This is also something that could be marketed by local Member Associations.
Accountability IPPF should support Member Associations that have local board members or staff who have not signed up to IPPF principles, such as being pro-choice or supporting young people’s SRH rights. While it is important to ensure this is done sensitively and without undermining the stability of organisations, IPPF should use its leverage here. It is also essential that IPPF uses its influence to ensure money is well spent.
Continuum approach
Organisational development It may be necessary for IPPF to take a more hands-on approach to organisational development. Organisational development will be needed to ensure a range of skills noted in the review framework relate to the basics required to ensure effective delivery of programmes. MAs may need support or advice on issues such as human resources (including ‘youth-friendly’ in all job descriptions, supervision for youth-friendly services and for quality and safety; complaints procedures that take youth-friendly provision into account); data (collection, analysis and making it meaningful to staff); finance and fundraising, etc.
The above diagram illustrates a continuum of service provision such as: SRH services for young people only general SRH services that include a component for young
people Member Association services provided to all, with no specific
provision for young people other services, including state and private sector provision.
External evaluation of SALIN+ projects: Overview report 2010 43
Youth centres Yo u
th
f ri e
100% YF
Nu
e mb
rs
nd
a cc
li n
Government, independent, outreach SRH/generic services ess
ess
s in g
er v
i ce
s
Cost +++ YFS +++ Numbers served +
‘Essential’ standard YF services
Cost + YFS ++ Numbers served +++
Figure 2: Achieving a balance between comprehensive and essential youth-friendly services developed by the review team
Specific, targeted SRH services for young people have the potential to include all necessary components (and more) of youth-friendly services to meet the needs of young people. However these services can be costly and may only be able to ensure access to a relatively small audience.NGO, private sector and government services (including, for example pharmacies, which are used by a wide range of people, including in rural areas) may reach much wider audience, but may not be young people friendly. It may be useful to consider what balance of services will achieve maximum coverage for young people. Young people in this review identified a limited number of essential criteria for services to be considered youth-friendly: 1) confidentiality; 2) positive, nonjudgmental attitude; 3) involvement of young people). Programmes should work towards achieving these in the widest number of services, while maintaining specialist young people-only services and centres as models of excellence. The second model above shows a balance that can be achieved between these types of provision to ensure maximum availability of youth-friendly services. It is also helpful to have a single set of criteria that is agreed across all partners in a country. It may be helpful to have a set of IPPF guidelines that take into account the ‘essential criteria’ and take a flexible approach to individual country development and tailoring. If this model were adopted, it may be possible for organisations to market their skills in building youth-friendly service provision, training staff, offering peer educators as ‘mystery clients’ and other income generation possibilities.
The SALIN+ programme came at a time of transition for IPPF, with the move towards more restricted and outcomes-based funding. The programme brought with it additional challenge of being one-off, short term funding for an issue that was new for many areas. The review has highlighted challenges in relation to universal commitment of staff, service providers and board members to young people’s SRH rights; sustainability of interventions; consistent and sustainable increased access to services; capacity in relation to reporting, use of evidence in planning and reporting and sustainable, high quality peer education programme. At the same time the review has highlighted dynamic organisations with strong commitment to young people’s SRH and creative solutions to the challenges put before them – with successful advocacy for CSE; outreach and peer education to support reaching underserved groups; some strong systems and organisations. IPPF has an opportunity now to tackle the challenges and to build on the successes to date to ensure that at central, regional and local level young people – and their SRH rights – are at the centre of planning, service delivery and review.
44 External evaluation of SALIN+ projects: Overview report
External evaluation of SALIN+ projects: Overview report 2010 45
Annexes
Annex 1 IPPF SALIN+ Projects
45
Annex 2 SALIN+ – External Evaluation and ‘Getting the basics in place’ framework
46
Annex 3 Country summaries and ratings
56
Annex 4 IPPF Clinic service data
68
Annex 5 Review schedule
72
Annex 6 Young people’s training workshop
73
Annex 7 Young people’s questionnaire
77
Annex 8 SALIN+ external evaluation validation meeting
78
Annex 9 IPPF Terms of Reference for SALIN+ Evaluation
85
46 External evaluation of SALIN+ projects: Overview report 2010â&#x20AC;&#x201A;
External evaluation of SALIN+ projects: Overview report 2010 47
Annex 1 IPPF SALIN+ Projects
Evaluation countries in bold MA
Region
Project Title
Ghana
Africa
Comprehensive Reproductive Health Education and Services for Young People
Malawi
Africa
Making it work: Improving Access and Quality of “Youth-friendly” SRH and HIV Services in Malawi
Mozambique
Africa
For Youth: Increasing Access to SRH services in Mozambique
Morocco
Arab World
Provision of Youth-friendly SRH services
Palestine
Arab World
Access to quality SRH youth-friendly services in Palestine
Tunisia
Arab World
Strengthening of Youth-friendly SRH service provision for vulnerable young people in Tunisia
Albania
European Network
Increased and Utilized Youth-friendly Services in the three ACPD Centers
Bosnia
European Network
Promoting Sexual and Reproductive Health of Young People in Sarajevo Canton
Kyrgyzstan
European Network
“You are not alone!”
Malaysia
East & Southeast Asia
Strengthening Youth-friendly Services and Information
Philippines
East & Southeast Asia
Youth-friendly Services for Young People’s Sexuality in ASRH in the Philippines
Vietnam
East & Southeast Asia
Promoting youth-friendly services (YFS) to young people in Vietnam
Bangladesh
South Asia
Strengthening Youth-friendly Services in FPAB
India
South Asia
Strengthening Youth-friendly Services in FPAI
Pakistan
South Asia
Strengthening Youth-friendly Services in FPAP
El Salvador
Western Hemisphere
Strengthening and Scaling Up Youth-friendly Services
Guatemala
Western Hemisphere
Strengthening and Scaling Up Youth-friendly Services
Honduras
Western Hemisphere
Strengthening and Scaling Up Youth-friendly Services
48 External evaluation of SALIN+ projects: Overview report 2010
Annex 2 SALIN+ – External Evaluation and ‘Getting the basics in place’ framework SALIN+ External Evaluation Framework – overview 1 SALIN+ Universal objectives2 1 To strengthen, increase and expand access to youthfriendly sexual and reproductive health services for young people
1.1 Health based outcomes 1.2 Rights based outcomes
2 To increase institutional commitment to youth-friendly services & young people’s sexual rights
2.1 Institutional commitment
3 To promote the provision of comprehensive sexuality education and information to all young people
3.1 Comprehensive sexuality education (CSE) 3.2 IEC and Peer educators
4 To build supportive communities for the realisation of young people’s sexual rights
4.1 Advocacy 4.2 Involvement of parents and other gatekeepers 4.3 Community awareness
2 Enabling environment 5 Policy & planning
5.1 Policy environment 5.2 Plans
6 Information & evidence base
6.1 Needs assessment 6.2 Data and information 6.3 Effective and innovative
7 Partnership & engagement
7.1 Core partnerships 7.2 Wider partnerships 7.3 Civil society engagement
8 Capacity & resources
8.1 Job descriptions & recruitment 8.2 Training 8.3 Skills 8.4 Supervision 8.5 Resources
9 Communication IEC (see above, element 3)
9.1 Partners – external 9.2 Media 9.3 National network – staff/internal
10 Service delivery
10.1 Range – of services 10.2 Range – of clientele 10.3 Accessible, YP friendly services
11 Accountability
11.1 M+E 11.2 Lesson learning 11.3 Governance
3 Crosscutting elements (not rated) Sustainability Promising practice Added value
External evaluation of SALIN+ projects: Overview report 2010 49
Priority clientele Unaccompanied married/unmarried young women YP living with HIV/AIDS YP with disabilities (physical / mental) Young sex workers Un/married young men YP who have been sexually abused YP who are illiterate YP who are homeless / living on the street YP in school / out of school / in work YP who identify as lesbian, gay bisexual or transgender YP who abuse drugs and/or alcohol Others
50 External evaluation of SALIN+ projects: Overview report 2010
SALIN+ External Evaluation Framework 1 SALIN+ Universal objectives Rating: GS/MP/SP
Evidence
Comments
Action/ Responsible/ Timing
1 To strengthen, increase and expand access to youth-friendly sexual and reproductive health services for young people 1.1 Health based outcomes: The project has increased numbers of YP who receive a range of SRH services a The number of YP receiving SRH services has increased during the project (number/percentage) b The Member Association has been successful in reaching out to underserved YP c There is a wider range of services available to YP as a result of the project [see 10 below, list of services] d The facilities of the Member Association’s clinical services have improved as a result of the project and this has improved access e The Member Association’s non-clinical services have improved as a result of the project and this has increased access 1.2 Rights based outcomes: Project has increased rights based outcomes for YP. a YP have participated in the design, delivery and monitoring of the project b The project has addressed gender inequity c The project has increased access for young women d Underserved groups (YPLHIV/AIDS, sex workers, OSYP, disability, etc.) feel able to access services e Young clients recommend the MA’s services to other YP 1.3 Other issues: 2 To increase institutional commitment to youth-friendly services and young people’s sexual rights (See also Service delivery – 10) 2.1 Institutional commitment: a The project has been effective in changing the values of service providers in relation to YP’s sexual rights b The project has been effective in changing the values of Member Association staff members in relation to YP’s sexual rights
â&#x20AC;&#x201A; External evaluation of SALIN+ projects: Overview report 2010 51
c The Member Association has implemented a child protection policy, with impact on the Member Association/ service provision d The Member Association has implemented a youth policy and can demonstrate its impact on the Member Association/service provision e There is recognition within the Member Association of the barriers that YP face f There a commitment to addressing barriers faced by YP on an ongoing basis g The Member Association makes efforts to understand the concerns, realities and SRH issues facing YP within its communities and acts on this h The Member Association has made known that it is a youth-friendly organisation through communication strategies aimed at attracting YP and can demonstrate the success of these efforts 2.2 Other issues: 3 To promote the provision of comprehensive sexuality education and information to all young people 3.1 Comprehensive sexuality education (CSE) a The Member Association promotes comprehensive sexuality education in schools or other institutions that work with YP through the project & can demonstrate successes b The project promotes provision of CSE to YP & can demonstrate successes c The project can demonstrate that YP are more aware of their sexual rights as a result of the CSE promotion 3.2 Other issues: 3.3 Information, education and communication (IEC) and Peer educators a IPPF IEC standards b The messages communicated by the Member Association are in line with the IPPF IEC checklist c Peer educators: d Peer educators are equipped with enough SRH knowledge to conduct outreach with their peers e Peer educators trained and confident in communication strategies for conducting outreach with their peers f The project has identified the greatest challenges in the Member Associationâ&#x20AC;&#x2122;s peer education programme and the solutions taken to these. 3.4 Other issues:
Ask / Not rated
52 External evaluation of SALIN+ projects: Overview report 2010
4 To build supportive communities for the realisation of young people’s sexual rights 4.1 Advocacy a MA advocates for changes to laws or policies that limit YP’s access to SRH services or information & can demonstrate successful strategies/achievements [nationally and locally] b sees a role for itself in advocating for the above changes, and if not, can explain why c In relation to YP’s sexual rights, the Member Association is an organisation that a) exerts influence; b) resists social norms/practices/laws that act as barriers; OR c) acts as an agent of change 4.2 Other issues: 4.3 Involvement of parents and other gatekeepers a The project has been successful in involving parents and other gatekeepers, and can demonstrate strategies used and achievements b The project can demonstrate the impact the involvement of parents and other gatekeepers has had on YP’s access to SRH services and information 4.4 Community awareness a The project has addressed cultural, social and religious barriers and can demonstrate how successful interventions have been 4.5 Other issues:
External evaluation of SALIN+ projects: Overview report 2010 53
2 Enabling environment Rating: GS/MP/SP
5 Policy and planning 5.1 Policy environment: National policy is supportive to YP’s Ask / Not rated SRH services 5.2 Plans Services and activities are based on a plan with SMART3 targets, timeframes, resources and identified leads and is informed by data and evidence of effectiveness, developed with the involvement of YP and include a focus on young people. || overview || annual
strategy (MA)
action plan (MA/SALIN+)
5.3 Other issues: 6 Information and evidence base 6.1 Needs assessment: A needs assessment taking into account a range of target groups (females/males, SALIN+ age range, most disadvantaged groups) inform service design and delivery: MA 6.2 Data and information: Sex disaggregated data – including baselines – is collected a for planning, service delivery, and monitoring and evaluation of progress/ effectiveness of targeted programmes for all prioritised groups, including those most at risk/most marginalised (see above list of clientele) b in relation to human resources – e.g. numbers of staff & volunteers/pay levels/position/access to training etc. (M/F) c Capacity: There is staff capacity to analyse & use data 6.3 Effective and innovative: Project activities and services are based on evidence of effectiveness and are innovative in response to need 6.4 Other issues: 7 Partnership and engagement 7.1 Core partnerships: Effective partnerships including public sector, NGOs / CBOs are in place to deliver core programs and services
Evidence
Comments
Action/ Responsible/ Timing
54 External evaluation of SALIN+ projects: Overview report 2010
7.2 Wider partnerships: Partnerships are in place to deliver related programmes (e.g. police [e.g. to deal with domestic violence or sexual assault], education [e.g. to ensure young mothers can continue education], income generation schemes [e.g. for YP with no work, sex workers who want different employment], women’s shelters [e.g. for domestic violence]) 7.3 Civil society engagement: Communities, male/female service users/non-users, priority groups (see priority clientele list) in relation to YP SRH participate in needs assessment, planning, delivery and monitoring 8 Capacity and resources 8.1 Job descriptions and recruitment: Job descriptions include a requirement to be young people friendly and pro choice and recruitment of staff takes this into account. 8.2 Training: Project staff/volunteers receive training to ensure they provide effective, safe, YF SRH services 8.3 Skills: Project staff/volunteers have the necessary skills to plan and deliver effective, safe, YF SRH services 8.4 Supervision: Staff/volunteers are appropriately supported and supervised including in relation to effective, safe, YF SRH services 8.5 Resources: There are sufficient resources to deliver effective, safe and sustainable YF SRH services to the project’s clientele, including a Both male and female staff to allow clients choice of sex of service provider b Other? 8.6 Other issues: 9 Communication IEC (see above objectives 3) 9.1 Partners – external: Systems are in place for effective communication among partners and stakeholders 9.2 Media: The project makes efforts to ensure positive publicity in the media for YP’s SRH services (or can explain why this is not possible) 9.3 National network – internal: The project ensures effective communication with its staff and its network of services. 9.4 Other issues: Staff meetings 10 Service delivery (see also objectives 1-3, above) 10.1 Range – of services: A full range of services is provided to young people, including: a Contraception (pills, IUDs, injections, etc.)
â&#x20AC;&#x201A; External evaluation of SALIN+ projects: Overview report 2010 55
b Condoms c Emergency contraception (promoted in the event of general contraceptive failure rather than only for rape) d Testing and treatment for HIV and sexually transmitted diseases e Pregnancy testing f Pre-test counselling and counselling and support for those who test positive g Maternal care h Treatment for other gynaecological matters i Services for young people who have experienced sexual, physical or emotional violence j Abortion services k Pre and post abortion care l Treatment for male sexual and reproductive health concerns m Voluntary counselling services for YP who wish to talk about their sexuality, relationships, violence, pregnancy outcomes or any other issue n HPV vaccine o Smear tests 11 Range â&#x20AC;&#x201C; of clientele (targeted) a Un/accompanied unmarried young women b Un/accompanied young married women c Un/married young men d YP in school e YP out of school f YP living with HIV/AIDS g YP who have been sexually abused h YP who identify as lesbian, gay bisexual or transgender i YP with disabilities (physical / mental) j YP who are illiterate k YP who misuse drugs and/or alcohol l Young sex workers m Working (see also illiteracy, below) n Homeless, living on the street
56 External evaluation of SALIN+ projects: Overview report 2010
o Rural p Other 11.1 Accessible, youth friendly services a Accessibility / Publicity || YP know about the service & what it offers through a range of publicity sources || The location is easy to find and has good links to public transport || Opening hours are appropriate for YP’s needs, offering both drop-in and appointments || Outreach to marginalised and vulnerable groups of YP b Confidentiality || A clear policy on confidentiality is in use || Staff are fully trained on the policy and its use || Confidentiality policy & practice fully communicated to YP || Reception staff are aware of the need for confidentiality c Attitude: Staff values and attitudes are positive in relation to YP’s rights and SRH in line with IPPF guidelines d Joined-up working (linked to Partnership & engagement – 7) || There is good inter-clinic communication to support YP accessing services || Partnerships in place with other YP service providers and appropriate referral pathways in place e Responsive services || A range of feedback mechanisms are in place, such as suggestion box, graffiti board, complaints procedure, ‘mystery client’, etc. that are known to young people using services. || Protocols/procedures are in place to act upon complaints in an expedient manner. Complaints are either acted upon or there is an explanation why not. 12 Accountability 12.1 M+E: Systems are in place for formal monitoring and evaluation of the project, and YP are involved in the M+E 12.2 Lesson learning: Information gathered through M+E and wider review are accessible/disseminated for lesson learning 12.3 Governance: There is formal governance in place to address issues arising and includes input or involvement by YP and there is a gender balance in membership Other issues:
External evaluation of SALIN+ projects: Overview report 2010 57
3 Crosscutting issues – Sustainability, promising practice and added value Sustainability 1 Which project interventions and/or activities have been incorporated into the MA’s annual work plan for 2011?
2 Has the Member Association made an effort from the beginning of the project to conduct activities in the most costefficient way? 3 What challenges will the Member Association face in sustaining essential project activities without further restricted funding? 4 Will the MA have to cut staff after the SALIN+ funding ends? If so, what impact will this have on services for YP? Promising practice 5 What intervention(s) have been most successful in increasing the number of YP accessing services and information? How do you know? What are the three most promising practices? 6 What have been the most successful strategies for reaching out to underserved YP? 7 What 3 promising/good practices would the Member Association like to share from its experience under SALIN+? Added value 8 Are there any results from the project that were unexpected? If so, provide an explanation of what they are and what their impact was/is. 9 How is the Member Association contributing to the strengthening of the national health systems and youth-friendly service provision in the public sector? 10 How has project impacted on the Member Associations’ data collection mechanisms? Anything else
Questions will be tailored to the interviewee, and will focus on the following:
Getting started (1): There is little or nothing in place to meet
the element Making progress (2): Substantial progress, but gaps remain in
What went well – the main successes (were they anticipated)?
What were the main challenges? What are three key recommendations for the Member Association in taking forward YP SRH services Was this (short term-funded) project ‘worth it’ in terms of service provision for young people?
some areas. If arrangements are not in place, there are plans to ensure improvement Systematically in place (3): Fully incorporated across the programme. While not all elements can be in place at all times, there is an overall sense that the Member Association is working at that level of provision and that there are systematic arrangements to ensure delivery.
58 External evaluation of SALIN+ projects: Overview report 2010
Annex 3 Country summaries and ratings
Albania 1 SALIN+ Universal objectives2 1 To strengthen, increase and expand access to youthfriendly sexual and reproductive health services for young people 2 To increase institutional commitment to youth-friendly services & young people’s sexual rights 3 To promote the provision of comprehensive sexuality education and information to all young people 4 To build supportive communities for the realisation of young people’s sexual rights
2
1.1 Health based outcomes 1.2 Rights based outcomes
2 2+
3
2.1 Institutional commitment
3
2+
3.1 Comprehensive sexuality education (CSE) 3.2 IEC and Peer educators 4.1 Advocacy 4.2 Involvement of parents and other gatekeepers 4.3 Community awareness
2+ 2+ 32-
5.1 Policy environment
-
5.2 Plans
3
6.1 Needs assessment
3
6.2 Data and information
2
6.3 Effective and innovative
2+
7.1 Core partnerships
3
7.2 Wider partnerships
2+
7.3 Civil society engagement
2+
8.1 Job descriptions & recruitment
3
8.2 Training
3
8.3 Skills
3
8.4 Supervision
3-
8.5 Resources
2-
9.1 Partners – external
3
9.2 Media
2+
9.3 National network – staff/internal
3
10.1 Range – of services
2
10.2 Range – of clientele
2
10.3 Accessible, YP friendly services
2+
11.1 M+E
2+
11.2 Lesson learning
2+
11.3 Governance
3
2+
2-
2 Enabling environment 5 Policy & planning
3
6 Information & evidence base
2+
7 Partnership & engagement
8 Capacity & resources
9 Communication IEC (see above, element 3)
10 Service delivery
11 Accountability
3-
2+
2+
2
3-
External evaluation of SALIN+ projects: Overview report 2010 59
Summary findings and conclusions The SALIN+ programme has been a considerable success for ACPD. It has enabled the organisation to develop its services to young people through a programme that involved development and refurbishment of centre and clinic space, training and integration of young people’s issues into well-established programmes and partnerships. Review interviews suggest high levels of satisfaction across the programme from a range of service providers and users as well as other stakeholders – parents, teachers and NGOs. ACPD is well known and well regarded and it appears that the SALIN+ project has added to that positive image. In terms of the four universal objectives, there have been increases in health and rights related outcomes and a strengthening in IEC, peer education and advocacy. ACPD has very strong systems to ensure that the ‘basics are in place’ and the organisation rated well across all these elements. Key strengths have been clinical infrastructure development including an entirely new clinic and centre in Tirana which was open for 8 months of the project; training and providing values clarification for all staff, board members/peer educators; establishing peer educators and other youth involvement mechanisms; advocacy to support comprehensive sexuality education (CSE); improved data collection; strong partnerships; increased access for young people in schools and universities as well as building on work with Roma and Egyptian young people; young people being heard on the board of ACPD. Challenges have cut across elements and have included: Service uptake: By far the biggest challenge ACPD faces is
to sustainably scale up to ensure more young people use its services. The current service configuration means there are limited opportunities for non-clinical services and there is limited outreach. (element 1) The need for transition from women’s services: As an organisation that had previously served primarily women, ACPD still has some way to go before it understands how to ensure young men are making use of services. (elements 1, 8, 10) Targeting: ACPD was wise to focus on an achievable target group (young people in schools/universities), but there is much more to be done to ensure access by marginalised young people across Albania. (elements 1,10) Work with parents and communities: While ACPD had some successes with teachers, more work is needed here. (element 4) Sustainability: Funding is not assured for future provision of SRH services for young people. ACPD does not currently bring in enough funds from other services to support provision of low cost to free services for young people most in need. (elements 1, 10, crosscutting)
Communication: There is not yet a means of communication
for young people across Albania and more work is needed with a range of media to publicise ACPD’s services and debate issues of interest to the organisation and its clients (and potential clients). Information and best practice: ACPD collects a wealth of data and information, but does not use it as fully as it might. (elements 3, 9) Quality assurance and retention of young people: Attracting and retaining young people as both centre users and as peer educators and ensuring the quality of their input will be an ongoing challenge for ACPD. (elements 3, 8, 10) Best use of resources: The young people who have participated as peer educators have made a tremendous contribution to SALIN+. However, they could do even more. (elements 1, 3, 4, 10)
Recommendations A number of recommendations are made, starting with the need to keep up – and build on – the good work. ACPD has put in place many key ingredients. To be able to maintain momentum and improve quality with reducing resources it will important to be rigorous about identifying key factors for ACPD’s successes and rolling them out. Issues include: Increased access to services: The main recommendation is to
significantly increase sustainable access to services both to those in ACPD’s current service areas and more widely across Albania (elements 1, 10) Targeting: More, effective targeting of young men and young people who are poor, rural, working and illiterate, as well as building on targeted programmes currently in place. (elements 1, 3, 4, 10) Making better use of young volunteers: For example in national advocacy, as youth counsellors and in monitoring and evaluation (elements 1, 3, 4, 10) Transition from women’s service provider: It will be important to develop effective ways of attracting, engaging with and providing services to young men. (elements 1, 8, 10) Sustainable programmes and making the most of available resources (element 1, 3, 4, 10) Integrated continuing professional development (CPD) (element 8) More work with parents and communities (element 3) Communicating for more effective services (element 3, 9)
60 External evaluation of SALIN+ projects: Overview report 2010
Bangladesh 1 SALIN+ Universal objectives2 1 To strengthen, increase and expand access to youthfriendly sexual and reproductive health services for young people 2 To increase institutional commitment to youth-friendly services & young people’s sexual rights 3 To promote the provision of comprehensive sexuality education and information to all young people 4 To build supportive communities for the realisation of young people’s sexual rights
2+
1.1 Health based outcomes 1.2 Rights based outcomes
2+ 2+
2
2.1 Institutional commitment
2
2-
3.1 Comprehensive sexuality education (CSE) 3.2 IEC and Peer educators 4.1 Advocacy 4.2 Involvement of parents and other gatekeepers 4.3 Community awareness
22 2 2+
5.1 Policy environment
-
5.2 Plans
3-
6.1 Needs assessment
3-
6.2 Data and information
3/1+
6.3 Effective and innovative
2-
7.1 Core partnerships
3 local/ 1 nat
7.2 Wider partnerships
2 local/ 1 nat
7.3 Civil society engagement
3 local/ 1 nat
8.1 Job descriptions & recruitment
1+
8.2 Training
2
8.3 Skills
2-
8.4 Supervision
2-
8.5 Resources
1+
9.1 Partners – external
3 local/ 1 nat
9.2 Media
2-
9.3 National network – staff/internal
2+ local/ 2 national
10.1 Range – of services
3
10.2 Range – of clientele
2
10.3 Accessible, YP friendly services
2
11.1 M+E
3
11.2 Lesson learning
2-
11.3 Governance
1+
2
2+
2 Enabling environment 5 Policy & planning
3-
6 Information & evidence base
2
7 Partnership & engagement
8 Capacity & resources
9 Communication IEC (see above, element 3)
10 Service delivery
11 Accountability
2+
2
2+
3
2
External evaluation of SALIN+ projects: Overview report 2010 61
Summary findings and conclusions FPAB and the SALIN+ project have done a tremendous job in a very short time – 18 months – of developing and implementing innovative, widely known and accepted youth-friendly (YF) centres – the tarar melas.23 Involvement of young people in service delivery through the tarar melas has been particularly valued, especially peer educators, youth counsellors and youth researchers. The government is not allowed to provide sexual and reproductive health (SRH) services for unmarried young people, but encourages NGOs to do so. FPAB is to be congratulated for carrying out this work in light of the government’s somewhat conflicting position on this important area. Review interviews suggest high levels of satisfaction across the programme from a range of service providers and users as well as other stakeholders – parents, teachers, NGOs. There was virtually universal support of the information and support function of tarar melas (though concern and resistance on the part of a few of access by unmarried young people to clinical services). Peer educators and other young interviewees were certain that young people who have been in touch with the tarar melas and related clinic services are more aware of their SRH rights as a result of SALIN+. FPAB is well known and well regarded and it appears that the SALIN+ project has added to that positive image. There have been increases in health and rights related outcomes and a strengthening in IEC, peer education and local advocacy and engagement. The project has been groundbreaking at reaching young people in schools and universities and providing information and access to YF services. There has been training for staff in the clinics associated with the tarar melas and staff are positive about sexual and reproductive health (SRH) rights of young people and providing YF services. FPAB has strong systems to ensure the ‘basics’ are in place. There are a number of areas that have been particularly well rated, including planning, needs assessment, local partnerships, range of services and civil society engagement. FPAB has made great progress in its work with and for young people at local level, where key stakeholders such as parents and teachers are impressed with the tarar melas. Parents in particular are strong in their support young people’s access to both information and services. Early in the review it became clear that one of the key issues for staff, parents, teachers, board members was the difference between married and unmarried young people in terms of their SRH rights. Currently there is a lack of consistency in FPAB’s services across the country – for example, in terms of commitment to unmarried young people’s SRH rights and also staff skills to deliver
YF services. Concerns were raised that there was not enough training for staff, peer educators or other youth workers and that quality assurance systems did not always focus strongly enough on whether services were youth-friendly. National advocacy for young people’s SRH rights was not a FPAB priority during the period of SALIN+, which may not be a concern in the short term. However national activities – partnership, advocacy, communication will need attention in the longer term, especially now that FPAB is increasingly able to demonstrate the benefits of providing information and services to young people in its tarar mela service areas. One-off funding with no assurance of continued resources can be a difficult challenge for the organisations tasked with delivery. Sustainability of programmes developed will be a challenge for FPAB and cost effectiveness of interventions will need to be more of a focus. While limited targeting – of young people in schools and colleges – is entirely reasonable in a short term project, it will be important to extend networks and partnerships to ensure a fuller range of IPPF priority groups are reached by FPAB services, including more boys and young men, rural young people, working young people, those who are not literate and others in IPPF’s priority populations not already using the services.
Recommendations Recommendations include maintaining momentum – keeping up and extending the excellent work currently underway. However, more will be needed to ensure the approaches undertaken are effective and cost effective. A further recommendation identifies ways to ensure consistent approaches to communication and service delivery – including young people friendly services – across the organisation. Specific recommendations address points in the framework covering areas such as: establishing and maintaining a balance between model YF
services and wider population access targeting priority groups national level action consistent approach across FPAB in relation to young people’s
SRH and YF services integrated continuing professional development (CPD) accountable decision-making sustainable programmes and making the most of available
resources communicating for more effective services
23 ‘Gathering of stars’, the name for the youth centres developed as part of the SALIN+ programme. See section 1.1 for details.
62 External evaluation of SALIN+ projects: Overview report 2010
Guatemala 1 SALIN+ Universal objectives2 1 To strengthen, increase and expand access to youthfriendly sexual and reproductive health services for young people 2 To increase institutional commitment to youth-friendly services & young people’s sexual rights 3 To promote the provision of comprehensive sexuality education and information to all young people 4 To build supportive communities for the realisation of young people’s sexual rights
2+
1.1 Health based outcomes 1.2 Rights based outcomes
2 3-
2+
2.1 Institutional commitment
2+
2+
3.1 Comprehensive sexuality education (CSE) 3.2 IEC and Peer educators 4.1 Advocacy 4.2 Involvement of parents and other gatekeepers 4.3 Community awareness
2+ 33 2
5.1 Policy environment
-
5.2 Plans
2+
6.1 Needs assessment
2
6.2 Data and information
2+
6.3 Effective and innovative
2+
7.1 Core partnerships
2+
7.2 Wider partnerships
2-
7.3 Civil society engagement
2
8.1 Job descriptions & recruitment
2
8.2 Training
2
8.3 Skills
2-
8.4 Supervision
2
8.5 Resources
2
9.1 Partners – external
2+
9.2 Media
3-
9.3 National network – staff/internal
2
10.1 Range – of services
2+
10.2 Range – of clientele
2
10.3 Accessible, YP friendly services
2+
11.1 M+E
2
11.2 Lesson learning
2-
11.3 Governance
3-
3-
2+
2 Enabling environment 5 Policy & planning
2+
6 Information & evidence base
2+
7 Partnership & engagement
8 Capacity & resources
9 Communication IEC (see above, element 3)
10 Service delivery
11 Accountability
2
2
2+
2+
2
External evaluation of SALIN+ projects: Overview report 2010 63
Summary findings and conclusions: APROFAM and the Mesoamerica project have done a tremendous job in a very short time – 18 months – of developing and implementing young peer programmes and delivering high quality services. Improvements have been made to facilities (including the new young people’s services in Guatemala City). Exit interviews suggest high levels of satisfaction across the programme. APROFAM is well known and well regarded and it appears that the Mesoamerica project has added to that positive image. There was widespread agreement among all participants in the review that advocacy and the development and work of the multiplicadores (peer educators) have been the greatest successes of the past two years. It is clear that advocacy has been a particular strength and success of the programme, with sex education made part of the national curriculum, and the multiplicadores and APROFAM gaining national credibility. The most notable challenge faced by APROFAM will be to continue to deliver the best of the results that have been achieved consistently across the organisation, country and with more partnerships and to more targeted audiences. Inconsistency was noted in terms of: Awareness and implementation of the youth policy and a
positive strategic attitude to young people and their rights to SRH services Delivery of the full range of services to young people (such as emergency contraception) Targeting of services to young people who need it most, including boys and young men, rural young people, people who are not literate and others in IPPF’s priority populations not already using the services. The table above shows the ratings given by the review team. Appendix 3 shows the results of the validation workshop and notes where there was discussion about the rating. When considering the ratings shown in the table above, it should be noted that APROFAM received a number of ratings of ‘systematically in place’ (as well as a few ‘getting started’).
Recommendations Recommendations include maintaining momentum – keeping up and extending the excellent work currently underway. However, more will be needed to ensure the approaches undertaken are effective and cost effective. A further recommendation identifies ways to ensure consistent approaches to communication and service delivery – including young people friendly services – across the organisation. Specific recommendations address points in the framework and suggestions include: Making more use than currently of all of the data collected,
to identify gaps and opportunities for greater reach and more targeted approaches Revising and marketing APROFAM’s IEC materials Maintaining high levels of quality of the multiplicadores Building on current strong partnerships to develop a more strategic partnership with Health, engagement of parents and other stakeholders, and wider partnerships to help young people make positive life choices A final recommendation suggests that APROFAM consider possible targeting of priority groups and what mix of young people’s service provision is developed to ensure coverage across the country.
64 External evaluation of SALIN+ projects: Overview report 2010
Mozambique 1 SALIN+ Universal objectives2 1 To strengthen, increase and expand access to youthfriendly sexual and reproductive health services for young people 2 To increase institutional commitment to youth-friendly services & young people’s sexual rights 3 To promote the provision of comprehensive sexuality education and information to all young people 4 To build supportive communities for the realisation of young people’s sexual rights
2+
1.1 Health based outcomes 1.2 Rights based outcomes
32+
2
2.1 Institutional commitment
2
2
3.1 Comprehensive sexuality education (CSE) 3.2 IEC and Peer educators 4.1 Advocacy 4.2 Involvement of parents and other gatekeepers 4.3 Community awareness
2 2+ 2 2
5.1 Policy environment
-
5.2 Plans
2
6.1 Needs assessment
2
6.2 Data and information
2-
6.3 Effective and innovative
2
7.1 Core partnerships
3-
7.2 Wider partnerships
2+
7.3 Civil society engagement
2
8.1 Job descriptions & recruitment
2
8.2 Training
2
8.3 Skills
2
8.4 Supervision
2-
8.5 Resources
2-
9.1 Partners – external
2+
9.2 Media
2+
9.3 National network – staff/internal
2
10.1 Range – of services
2+
10.2 Range – of clientele
2+
10.3 Accessible, YP friendly services
2
11.1 M+E
2+
11.2 Lesson learning
2
11.3 Governance
2+
2
2
2 Enabling environment 5 Policy & planning
2
6 Information & evidence base
2
7 Partnership & engagement
8 Capacity & resources
9 Communication IEC (see above, element 3)
10 Service delivery
11 Accountability
3-
2
2+
2+
2+
External evaluation of SALIN+ projects: Overview report 2010 65
Summary findings and conclusions
Recommendations
The AMODEFA SALIN+ programme aimed to:
Recommendations include maintaining momentum – keeping up and extending the excellent work currently underway. However, more will need to be done to ensure the approaches undertaken are effective/ cost effective and sustainable. Specific recommendations include:
establish and expand youth-friendly services across AMODEFA contract professional staff for youth-friendly service delivery
(doctors, nurse, psychologist, 3 youth coordinators, driver) train service providers and laboratory technicians produce behaviour change and communication (BCC) materials
and promotion of radio and TV programmes advocating young people’s sexual and reproductive health rights (SRHR) and access to care increase involvement of young men strengthen partnerships with other NGOs, public and private sector youth organisations develop microfinance activities to support young people living with HIV/AIDS (PLWA) and empower more vulnerable young women procure clinical equipment and mobile vans. AMODEFA has done an impressive job in a very short time – 18 months – of extending the delivery of young peer programmes and the delivery of quality youth-friendly services. Building on a long-term partnership with the government sexual health youth programme, Programa Geração Biz24 (known in this report as ‘Biz Generation’), they have ensured positive developments and created new partnerships and work in rural areas. Mozambique has vast social and health needs, specifically related to high levels of poverty and high rates of HIV/AIDS, with women being the most affected. This is the background to the establishment by the government of the ‘Biz Generation’ programme and the involvement of key partners like AMODEFA. This is a unique programme in Africa.
be strategic – carry out a full needs assessment (internal and
external) to form the foundation of a strategic plan, identifying priorities for work over the next 5 – 10 years meet the unmet needs of boys and young men and other marginalised groups expand and develop the training programme to all AMODEFA staff (paid and unpaid) in relation to young people’s SRHR rights and needs, values clarification and monitoring and data needs and processes share the learning with partners of the SALIN+ programme and incorporate the learning into ongoing strategic developments advocacy to government on issues such: abortion; more sex education in the national curriculum; provision for continuing education for young pregnant women. In the midst of massive social and health needs, AMODEFA has a strong history of offering needed services to a range of young people. SALIN+ has enabled them to extend and enhance this work, including developing new services. There is a wellestablished organisational ethos of working with young people but there are challenges with the ending to SALIN+ and the loss of the funds. It is important to now be strategic and take the time to use the lessons learned from SALIN+ to plan strategically and identify new opportunities.
Key successes include: work with young people in very remote rural areas enhancement of the clinic provision extension of the peer education work and the targeted work
with the local deaf community partnerships with the ‘Biz Generation’ and partner organisations
are strong – working in schools, local communities and in the provision of youth-friendly services enhanced service delivery in Maputo and Namaacha with improved premises new clinic opened in Zambesia mobile clinic work in and around Maputo and in the remote areas surrounding Namaacha. use of dance and drama to engage young people – and often unexpected outcome of significantly improved data collection. 24 Programa Geração Biz is a Government programme running since 1999, operating in all 11 Mozambican provinces. The programme has 3 key components: school-based work with counselling corners developed in close collaboration with the Ministry of Education; a community-based programme, with youth centres under the leadership of the Ministry of Sport and Youth; and youth-friendly clinic services based within Ministry of Health facilities. The name literally means “Biz Generation” and was chosen by young people.
66 External evaluation of SALIN+ projects: Overview report 2010
Philippines 1 SALIN+ Universal objectives2 1 To strengthen, increase and expand access to youthfriendly sexual and reproductive health services for young people 2 To increase institutional commitment to youth-friendly services & young people’s sexual rights 3 To promote the provision of comprehensive sexuality education and information to all young people 4 To build supportive communities for the realisation of young people’s sexual rights
2+
1.1 Health based outcomes 1.2 Rights based outcomes
2 2+
2+
2.1 Institutional commitment
2+
2
3.1 Comprehensive sexuality education (CSE) 3.2 IEC and Peer educators 4.1 Advocacy 4.2 Involvement of parents and other gatekeepers 4.3 Community awareness
2 22+ 2
5.1 Policy environment
-
5.2 Plans
2+
6.1 Needs assessment
2
6.2 Data and information
2
6.3 Effective and innovative
2
7.1 Core partnerships
2+ local/ 1+ nat
7.2 Wider partnerships
2+ local/ 1 nat
7.3 Civil society engagement
2
8.1 Job descriptions & recruitment
2-
8.2 Training
2+
8.3 Skills
2
8.4 Supervision
2-
8.5 Resources
2-
9.1 Partners – external
2+ local/ 1 nat
9.2 Media
2-
9.3 National network – staff/internal
2-
10.1 Range – of services
2
10.2 Range – of clientele
2+
10.3 Accessible, YP friendly services
2
11.1 M+E
2
11.2 Lesson learning
2
11.3 Governance
2
2
2
2 Enabling environment 5 Policy & planning
2+
6 Information & evidence base
2
7 Partnership & engagement
8 Capacity & resources
9 Communication IEC (see above, element 3)
10 Service delivery
11 Accountability
2+
2
2
2
2
External evaluation of SALIN+ projects: Overview report 2010 67
Summary findings and conclusions The Yes4Yes SALIN+ programme aimed to: refurbish four centres and develop youth-friendly services; increase the numbers of young people accessing services (outreach, clinical, counselling and educational); recruitment and training of youth volunteers to work as educators and counsellors; advocacy on both a local and national level; greater awareness of the youth-friendly services of FPOP. FPOP and the Yes4Yes project have done a tremendous job in a very short time – 18 months – of developing and implementing young peer programmes and delivering quality youth-friendly services. With positive leadership, FPOP have been able to develop and respond to local needs, and developing strong local partnerships. As an organisation FPOP has incorporated the needs of young people into all strategic planning and put them at the centre of future developments. The greatest achievement of the FPOP SALIN+ programme was embedding youth services and young volunteers into the whole of the organisation. For the first time FPOP has gained a reputation as experts in offering young people friendly services. A strong training programme has supported the volunteers and young people have been involved in many aspects of the Yes4Yes programme, as well as across FPOP as an organisation. Targeted work with marginalised groups of young people has been successfully meeting distinct local needs. All of FPOP’s work is developed in an environment not supportive of their approach due to religious and cultural beliefs that cause huge barriers to young people’s sexual and reproductive health (SRH) rights and access to services. The young people have been actively advocating, with the whole of FPOP, for the pending Reproductive Health (RH) Bill to enable their work to develop more and support both young people and the wider population access services. There were some inconsistencies noted and challenges in terms of: Delivery of the full range of services to young people (such as
emergency contraception, HIV testing) Targeting of services to boys and young men, people who are
not literate and others in IPPF’s priority populations not already using the services Inability to uniformly offer HIV tests due to Department of Health barriers. The table above shows a summary of the ratings given by the review team. Appendix 3 shows the results of the validation workshop and notes where there was discussion about the rating. When considering the ratings shown in the table above, it should be noted that there were some distinctions of what was achieved at local level versus what was achieved at national level. For
example, local partnership level was very well developed but at the national level lacked the necessary relationships with ministries and other potential national partners.
Recommendations Recommendations include maintaining momentum – keeping up and extending the excellent work currently underway. However, more will need to be done to ensure the approaches undertaken are effective/ cost effective and sustainable. Specific recommendations address some inconsistencies and include the suggestions to: Meet the unmet needs of boys and young men and other
marginalised groups who are not accessing services in large numbers Expand and develop the training programme to all FPOP staff in relation to young people’s SRHR rights and needs, values clarification and monitoring and data needs and processes Ensure all staff and volunteers are young people friendly and supportive of their sexual health rights and are pro-choice, irrespective of personal beliefs Develop a rolling programme of recruitment and training of the youth volunteers to ensure a core group is always available Develop partnerships at a national level with key ministries, international NGOs and national youth and sexual health programmes. This will require positive advocacy from the headquarters, as well as at Chapter level Ensure the development of information, education and communication (IEC) materials to use in outreach and services and advocating for comprehensive sexuality education as a part of the school curricula Further develop the model of working in partnership with the nursing college, which offers a positive way to ensure youthfriendly nurses and offers income generation for the Chapter Share the learning from the Yes4Yes programme and incorporate the learning into ongoing strategic developments. FPOP has established a strong foundation for the continuing development and extension of providing youth-friendly services and reaching some more marginalised groups of young people. The learning from the SALIN+ work should fully inform the future work across FPOP with young people and remain a strategic priority of the organisation.
68 External evaluation of SALIN+ projects: Overview report 2010
Tunisia 1 SALIN+ Universal objectives2 1 To strengthen, increase and expand access to youthfriendly sexual and reproductive health services for young people 2 To increase institutional commitment to youth-friendly services & young people’s sexual rights 3 To promote the provision of comprehensive sexuality education and information to all young people 4 To build supportive communities for the realisation of young people’s sexual rights
2
1.1 Health based outcomes 1.2 Rights based outcomes
2 2
2
2.1 Institutional commitment
2
2-
3.1 Comprehensive sexuality education (CSE) 3.2 IEC and Peer educators 4.1 Advocacy 4.2 Involvement of parents and other gatekeepers 4.3 Community awareness
221+ 2-
5.1 Policy environment
-
5.2 Plans
2-
6.1 Needs assessment
2-
6.2 Data and information
2
6.3 Effective and innovative
2-
7.1 Core partnerships
2
7.2 Wider partnerships
2
7.3 Civil society engagement
2-
8.1 Job descriptions & recruitment
2-
8.2 Training
2-
8.3 Skills
2-
8.4 Supervision
1+
8.5 Resources
2-
9.1 Partners – external
2-
9.2 Media
2-
9.3 National network – staff/internal
1+
10.1 Range – of services
2
10.2 Range – of clientele
2
10.3 Accessible, YP friendly services
2
11.1 M+E
2-
11.2 Lesson learning
2-
11.3 Governance
2
2-
2-
2 Enabling environment 5 Policy & planning
2-
6 Information & evidence base
2-
7 Partnership & engagement
8 Capacity & resources
9 Communication IEC (see above, element 3)
10 Service delivery
11 Accountability
2
2-
2-
2
2-
External evaluation of SALIN+ projects: Overview report 2010 69
Summary findings and conclusions
Recommendations
ATSR was extremely ambitious with the SALIN+ programme. In the two-year programme they aimed to support the delivery of youthfriendly services by:
Recommendations include maintaining momentum – keeping up the work currently underway. However, more will need to be done to ensure the approaches undertaken are effective/ sustainable, more effective and cost effective.
Upgrading seven existing ATSR centres to enable more focus on
youth services Opening three new youth centres, particularly based in the
Specific recommendations are focused on getting the processes in place and include:
north and poorest areas Be strategic – carry out a full needs assessment (internal and
The centres aimed to provide adequate education, information and orientation to young people of both sexes. Each centre recruited midwives,25 social workers and coordinators.
Summary findings and conclusions ATSR has done a positive job in a politically challenging and short period of time. There have been many barriers – cultural, time, size of country and the ‘Revolution’ – which have impacted on their work to achieve all of the objectives. The Revolution particularly impacted on the work for 2 – 3 months of the final months of the project and left a legacy of fewer young people accessing the services. The SALIN+ programme has enabled the refurbishment of seven centres and the opening of three new youth centres across Tunisia, as well as the employment of new staff. The centres are youthfriendly, equipped with computers and Internet access. ATSR has developed partnership work, particularly on a local level. In each centre a core of peer educators was recruited, though retention has been a key issue in some centres. Overall the work focused on information and education, with some clinical services available. ATSR was possibly overly ambitious in their aim to work across ten centres and not concentrate on establishing a smaller number of centres as models of good practice to share the learning from. The ability to sustain this level of service delivery is limited. Significant amounts of one-off funding, like SALIN+, with no assurance of continued resources can be a difficult challenge for any organisations tasked with delivery, particularly when not all of the systems are in place.
external) to form the foundation of a strategic plan, identifying priorities for work over the next 5 – 10 years Research and develop a strategic fundraising plan Meet the unmet needs of other marginalised groups who are currently not accessing services Ensure all staff and volunteers are young people friendly and supportive of their sexual health rights and are pro-choice, through implementation of policies and procedures for recruitment, training, and service delivery (including outreach) Establish an organisational training strategy for staff, young people and volunteers, including board members at regional and national level Develop a programme of recruitment and training of the peer educators, with appropriate support and supervision Share the learning from the SALIN+ programme and incorporate the learning into ongoing strategic developments Advocacy to government and with external partners on issues such: young people’s SRH rights; comprehensive sexuality education; gender equity, etc. Despite the challenges, ATSR has achieved the enhancement of services for young people and laid the foundations for ongoing service developments for young people. SALIN+ has enabled them to extend and enhance their work, including the development of new services. There are challenges with the ending to SALIN+ and the loss of the funds that enabled this new work. It is important to now be strategic and take the time to use the lessons learned from SALIN+ to plan strategically and identify new opportunities.
There are examples of positive practice: designing of a SRH monopoly game (in Arabic) (with the potential to market). In work with young deaf people, they identified a gap in sign language to adequately speak of sexual health issues – a project opportunity to work with them and develop further signs for greater discussion and education.
25 In Tunisia midwives are trained to work with all gynaecological needs of women not just childbirth
70 External evaluation of SALIN+ projects: Overview report 2010
Annex 4 Review schedule
For details of country review interviews, please see individual country reviews. Key dates and contacts include: Country (Member association)
Reviewer(s)
Dates
Key Contacts
Albania ACPD
B James
3–9 May 2011
Elona Hoxha Gjebrea (egjebrea@acpd-al.org) – Executive Director Brunilda Hylviu (brunahylviu@gmail.com) – Youth Coordinator
Bangladesh FPAB
B James
3–11 April 2011
Moazzem Hossain; Israt Jahan Baki – Youth Coordinator
Guatemala (pilot) APROFAM
B James K Watson S Daniel
7–14 March 2011
Sergio Penagos – Executive Director Dalila de la Cruz – Youth Programme Coordinator Edy Cruz – Coordinator youth-friendly services Erick Lemas – Youth worker
Mozambique AMODEFA
S Daniel
18–26 April 2011
Cecilia Bilale – Executive Director Claudia SImbine – Programme Coordinator Estevao Marrime – Youth Coordinator
Philippines FPOP
S Daniel
28 March – 5 April 2011
Roberto Ador – Executive Director Gessen Rocas – Programme Director Brayant Gonzales – Youth Coordinator
Tunisia ATSR
S Daniel
9–17 May 2011
Aicha Chaherli – Executive Director Ibtissem Jouini – Programme Coordinator Zohra Turki – National Board, Treasurer
External evaluation of SALIN+ projects: Overview report 2010 71
Annex 5 Young people’s training workshop Training workshop for young people supporting evaluation Aim of training To prepare and support local young SALIN+ peer educators / youth volunteers to participate in the evaluation process by carrying out interviews and focus groups with other young people and with stakeholders as appropriate. The training aims to support both the young people and strengthen the outcomes of the evaluation.
Introduction As an integral part of the external evaluation, we would like to involve local young people in the evaluation process. Given the limited timescale of the evaluation, extensive participation is not feasible but we aim to ensure a process to maximise their input. We value the unique input of the young people and the potential they have to hear voices that external evaluators may not hear or be told.
Support to youth volunteers The evaluators will pay all transport costs that the young person encounters to participate in the evaluation. Any other costs (e.g. childcare) will be individually negotiated and we hope we will able to cover. The SALIN+ staff will identify one person to support the volunteers during the time of the evaluation and the evaluators will be available to answer any questions but will not have time to provide day-to-day support.
Programme – 3 hours Timing
Activity
5 minutes
Introduction and background to evaluation
30 minutes
Participant introductions: Reporters 1 Pass out half a card to each participant 2 Ask participants to find the person with the other half of their picture 3 When the pairs are matched, ask them to take turns interviewing each other. Each person will have 3 minutes to learn the following things about his/her partner and be ready to introduce him/her to the whole group: i Name/ your work / school / what you’ve done with SALIN+ ii When you were a child, what did you want to be as an adult? OR iii One interesting thing about them that no one else knows 4 After 3 minutes ask them to swap who is being interviewed 5 Bring all back to the whole group after about 7 minutes. 6 Ask each participant to introduce their partner to the group – ensure they make it brief and to the point.
Pictures or postcards cut in
The evaluation – what’s it all about? What do we want? Introduce SALIN+ funding and need to evaluate what has been achieved with the funds. Key things we are looking for are: zz What has been developed zz What was good, what didn’t work zz People’s experiences of the services zz The impact on the sexual health of local young people, etc. Introduce the framework and process that will be happening during the week
Flipchart Marker pens
15 minutes
Resources needed
half Scrap paper Pens / pencils Flipchart Marker pens
72 External evaluation of SALIN+ projects: Overview report 2010
Timing
Activity
Resources needed
30 minutes
Your experiences of SALIN+ Gaining an understanding of their roles and experiences of the SALIN+ work. In small groups or pairs, get them to identify three things for each question: What has been really good? What could have been improved? What do you recommend for the future? Overall, was it worth it? Write 6 large sheets of paper with one question on each one and ask the young people to put each answer onto a post-it and put the post-its onto the large sheets.
Post-its Blu-tack
10 minutes
Ethics Introduce ethics and what it means in research: it is a code of conduct to guide and influence personal and professional behaviour and actions and should ensure: The autonomy and informed consent of the participants, No harm comes to participants but that they benefit from the process Equity for all involved. How should an interviewer behave and develop trust – general discussion to identify key requirements.
Flipchart Marker pens
15 minutes
Confidentiality Ask the participants to brainstorm their definition of CONFIDENTIALITY. Aim for the group to come to a consensus on the definition – it may be necessary to break them into smaller groups if there is disagreement, and then bring them back together to ensure a whole group consensus. Include in discussion the difference between anonymity (no names) and confidentiality (no gossip). Discuss what this means in context of them as young people interviewing other young people. Look at the importance of: zz Discussing confidentiality at the beginning zz Explaining what you mean – no names, nothing identifying – letters or pseudonyms zz Dangers of discussing what interviewees say with others Draw together discussion on the centrality of confidentiality with the research, in the services and the work of SALIN+.
Flipchart Marker pens
External evaluation of SALIN+ projects: Overview report 2010 73
Timing
Activity
60 minutes
Interviews A good interviewer is: An active listener, skilled in inter-personal communication Respectful of the respondent’s ideas and values Committed to learning from the interviewee Aware of the importance of non-verbal behaviours Good at drawing people out and in following up with gentle probing Skilled in taking notes / recording / remembering what was said. As part of the evaluation we want each young person to complete 2-5 (depending on numbers attending training) interviews in next 3 days. It would be good to interview both young people who have used the services, and if possible some who may not have used the services. Recording the interview – what and how? Think about how and what it is necessary to record and HOW Key ideas Specific positives & negatives – not a long story – the themes and issues If possible handwritten notes can be written up on the computer to give to the external evaluators. As a group decide how they are going to present feedback to the verification workshop (skit, talk, PPT presentation, written document – other ideas?) / or could give verbal feedback to someone who could write it up What definitely needs to be included in the written feedback, and in the presentation, is: Total numbers interviewed, ages, sex, occupation How many were positive about the YPF services How many were not positive about the YPF services What were the top three: zz Positives zz Negatives zz Recommendations What individually, and as a group, has been learned from doing the training and carrying out the interviews Your first interview The young people will interview each other Get them to do the interviews in pairs A general discussion on: What it felt like to be interviewed by a peer? What did it feel like to DO the interview? What were the best ways that the interviewer made the interviewee feel at ease / comfortable? Discuss how long an interview should take, the environment and other needs to do it successfully. Are there any other questions they think are missing and important? The information collected in the interview should be written up by the interviewer and given to the external evaluators.
Resources needed Questionnaire for
interviewing
74 External evaluation of SALIN+ projects: Overview report 2010
Timing
Activity
15 minutes
Closing – support, how they will feedback, final presentation at verification workshop (maximum 10 minutes. Reimburse any costs to get to training + for doing interviews. Share food with group – lunch or afternoon tea.
Resources needed
External evaluation of SALIN+ projects: Overview report 2010 75
Annex 6 Young people’s questionnaire
SALIN+ Questionnaire for young people Sex: Age: Occupation/Student: 1 In general, what do you think are young people’s sexual and reproductive health needs in XXX? 2 What do you know about the sexual and reproductive health services available for young people? 3 Have you used the XXX project? Or XXX services? If yes: i What services did you use? ii What was your experience like? iii How did you hear about the clinic? iv What were the 3 best things about the service? zz zz zz
v Is there anything you would like to change? What were the 3 key things to change? zz zz zz
vi What would you recommend for future young people’s sexual and reproductive health services? If no: vii Have you ever used other local sexual and reproductive health services? viii Do you know about any local services? 4 Where have you learned about sexual and reproductive health? 5 What do you think are the most important things for young people in relation to sexual and reproductive health – information/education, services, outreach work, counselling, etc.? 6 Is there anything else you would like to say? When writing notes please feel free to use the back of the sheet.
76 External evaluation of SALIN+ projects: Overview report 2010â&#x20AC;&#x201A;
Annex 7 SALIN+ external evaluation validation meeting Evaluation findings compilation (Note: there was no evaluation in Guatemala)
Albania
7
Time allocated for discussion
6
7
Overall assessment of the validation meeting
8
8
1
1
Approach
Were the findings and ratings accurate? Will the evaluation framework and findings be helpful for your work?
13
3
16
2
Poor
9
Not at all
Overall usefulness of discussions
Fair
8
Not very well
8
Fairly well
Very good
Overall quality/relevance of presentations
Completely
Excellent
Overall ratings
Good
10 May 2011 16 participants
External evaluation of SALIN+ projects: Overview report 2010 77
What was the most important thing you learned?
Bangladesh Total participants: 36 over the session Total number of evaluation forms: 30
Any other comments about the meeting or about the external evaluation generally? I don’t have any comments – we’ve said it all. Thank you. This worthwhile workshop will help me in my work a lot. Very professional and smart evaluator. It was very participatory, inspirational and reflective to future
work. It was such a fantastic experience the review and the workshop
work and projects together. Thank you. I think it is a very good analysis of what we are doing. With
responsibility. I want this project to be repeated. The meeting was useful and had very good ideas and
suggestions. Thank you.
Very good
Good
Fair
Poor
Overall quality/relevance of presentations
14
16
0
0
0
Overall usefulness of discussions
16
12
2
1
0
4
10
10
6
0
14
14
2
0
0
Time allocated for discussion
Approach
Not very well
Not at all
Overall assessment of the validation meeting
Fairly well
results of the review were positive, realistic. In a coalition I think it is important we get information about other programmes’ implementation in other countries. SALIN+’s collaboration was really ok That we should continue working in the undertaken road. Consideration of the basic elements The way this workshop was organised: taking feedback for the ratings you gave us was very constructive. How important it is to consult a wider number of young people How to find a way to offer services to max number of people – consider number of beneficiaries with low cost. Targeting for the future based on good fundamentals is such an important element and really promising for YFS in this country Referring to the SALIN+ Project, what I see is the importance and necessity of its theme in the Albanian reality The most important thing to me in this workshop was seeing the evaluation of the project because it was the first time and also seeing the progress the centre is making. Everything that has been discussed and achieved during this project has been very important. Evaluation of SALIN, recommendations brought. Sharing experiences All of the things were important
Overall ratings
Completely
suggestions Barbara was so nice! She made us feel really good. Generally the
Excellent
All the projects are developed considering young people’s
Were the findings and ratings accurate?
18
12
0
0
Will the evaluation framework and findings be helpful for your work?
25
5
0
0
What was the most important thing you learned? Liked the Youth Researchers’ work – that is very important. The
group work validating the findings was very good! It’s new for us! SALIN+ universal objectives and the framework Engaging Branch people very motivating Local fundraising interesting Partnership relations with other organisations; tarar mela vs. non tarar mela; beneficiaries evaluation (x2) Sustainability Findings were very good and that will be reflected in future program I learned scoring and assessing system with each other; sustainability Evaluation rating system was helpful to understand the project’s impact. And the participation of staff makes the work more meaningful. This method of evaluation is a new thing for me. Rating our own programme impartially as an implementer is a new and
78 External evaluation of SALIN+ projects: Overview report 2010
effective approach. Also involving young people as evaluator, interviewees proved effective and brought out information and recommendations that adults would not have been able to do so effectively. Learn about SALIN+, its objectives and findings of external evaluation of SALIN+ How to evaluate a project very effectively From this workshop specially I learned lot. This evaluation is so nice! I learn from here all the ideas of the SALIN+ projects. I’m the new member of FPAB family so SALIN+ and presentation is nice. About SALIN+ project Personal opinion of staff in review is important Coverage of YFS to general services How to ensure validation of study findings The way of comparison on the scores format through group discussion as well as general discussion I know about SALIN+ programme I learned most important thing is participation in group work – different points and presentation Rating of the objectives and other items Reaching to a point for a particular project with discussion by everyone. What project is doing to make it succeed or fail is important. Sustainability of projects – how to do it. The places have done well. Need to develop and make local ‘friends of tarar mela’ All things discussed, meeting has shown effectiveness How to speed up tarar melas and FPAB’s work SALIN+ + external evaluation framework – overview: Excellent subject.
Any other comments about the meeting or about the external evaluation generally? I say very good workshop. The workshop presenter very very
nice. I am very happy! In future this type of meeting needs day long time required and
participants from field will be double of this meeting. (x2) Overall evaluation done by external team is very good! Thanks to Barbara – very nice! Time allocated for meeting was short but topics of discussion
very good There was not enough time – we could have spent much more
time discussing these things! The evaluation captured all the main impacts It’s a good initiative. More young people and volunteers should
have been involved. It’s very good and meaningful, so I am very happy on this. The strategy of evaluating the project in many ways – it was
appreciable zz Involved YP in evaluation process
zz Shared the findings with all branch level manager, service
providers and staff members zz Methodologies adopted zz Tools used. Everything is ‘ok’ specially presentation and information also
‘choclets’ Need more time! The timing of this workshop is not very good. We need more
time to discuss these things. Timing is so so. It is useful to supply all the documents that we discussed in the
meeting Dissemination in the meeting was very focussed, clear and
concise. More youth participants from all over FPAB could be brought in this kind of meeting for information sharing. Thanks so much to facilitator for participatory evaluation process Should have been longer Need to implement recommendations from this meeting This type of meeting should be arranged more often. It made me feel very good! This kind of workshops should take place more than once in a year. See you again Barbara
External evaluation of SALIN+ projects: Overview report 2010 79
Mozambique evaluation workshop evaluation Do you feel today’s workshop has been useful? If yes, why? If no, why not? Yes it helped to understand the true dimension and of the
results of project SALIN.
How could the workshop have been improved? The classroom should have been the other one because it is
bigger still it was good. The assessment was well structured because it showed to
AMODEFA not to be relaxed and see that everything is not right.
It was good and it allowed a group discussion. For me the assessment was very helpful made me understand
I would like to say that the assessment would have been more
how important is group work and the results you can achieve working as a team. It showed me as well that in any organization or activity you will always need people to develop. The assessment was very helpful for me because we discuss about various cases to improve AMODEFA Yes it was because we learn about new things and we could evaluate the project much better. It gave us an insight on how to run the project much better. Yes because I learn a lot. The assessment was very useful and made me learn more deeply about the work we are doing and how to evaluate it. It was very productive. I learn about the difficulties and the necessities on our program and how can it be improved. Yes it has helped me understand the process of how our work is done in AMODEFA. Learn a lot. It was very useful and everyone agreed with the results. It was very useful. I learn on how to analyze my work skills.
The training would have been better if we had more funds. It was very good. Nothing to comment. For me was good however it missed some methods. It was great could not have been better. We should have more staff in all provinces We should do more assessments.
What is the most important thing you have learned today? The form the evaluation was made it involved everyone and
everyone contributed. Taught me how to assess my work and the importance of group
work. The classroom should have been better if they would have
requested better structure To pass information to other youth all over the country. I learn how to detect about cases that didn’t even occur to my
mind. The most important things I learn today was to work as a team and how my colleagues see problems. The recognition and assessment of our work. Been able to spread the word about our program in urban and rural areas. I learn that we need to always analyze our work and auto criticise our self to improve our work. I could change experiences with other people. Analyse and evaluate of objectives.
useful if it was 1 more days.
Has the overall evaluation process been beneficial to you and your work? It helped me understand that working as a team we can achieve
better results. It will help the work We will still keep the activities by SALIN. Yes it was. Yes it was. Yes Yes it was very useful. The process was great it will help me with my work. It was because I learn new techniques on how to improve some
of my skills, like on how to interview and teach people on our community. I learn a lot. It is good because it will help us with our programme in AMODEFA. Yes it was. It involved everyone. It was very useful and it helped me evaluate my work.
Any other comments? I would like to say thank you. The training will help me change
my ways of working. I would like to thank you for this opportunity that was given to
discuss about important matters in relation to youth. Nothing to say only it was a group work. To continue with this type of work because without the
comments we will not be able to see our faults and develop our work. When I always have the opportunity to learn I create the opportunity to teach as well.
80 External evaluation of SALIN+ projects: Overview report 2010
We should have more time one day is not enough. It was very good. They are intense sessions It was very useful and it helped me evaluate my work.
Philippines Feedback from Validation Workshop Attended by 28 people – National Office Staff, clinic staff, partners and young volunteers
Do you feel the workshop has been useful? If yes, why? If no, why not? Yes, based on my experience, I’ve learned how to interact with
other people, specially in urban areas Yes, so that we will also know as an organization our strengths and weaknesses (for its so important we know where to improve, what to improve and whatever to sustain Yes – the workshop shows the performance / achievements of SALIN+ Yes – because through this activity, we are able to know our performance in terms of goals/ objectives and accomplishments Today’s workshop was very useful because we get to review and evaluate the job we have done Yes coz it serves as a reminder and gives us a sense of direction. We’re able to see what are our strengths and what are the things we need to improve Yes – because we saw the pros and cons – what in the project we must improve The evaluation workshop today is very useful because I have realized my important role as a service provider on how to deliver an effective YFS properly Yes because I learned a lot of things that I can share with my friends and people around me The workshop is really useful for us advocates. It enlightens us more to be equipped on our future plans for the sustainability of the actions and commitments – not just to the organisation but with people we are serving to who really needs us Yes it was successful and useful because it helped us to evaluate ourselves and to know your feedback on our work for almost 15 months Useful in the sense that we were able to negotiate with ranking where we do not agree and/or clarify why we were giving such a ranking. We heard other chapters experiences Yee!... Yes – today’s workshop was very useful because we have addressed the full results of the implementation of the project and as well evaluate what is the needs to be improved, what can we do to sustain
Yes because we can get to know what to improve in our
different clinics Yes. I believe that today’s workshop has been useful because we
were able to evaluate the SALIN+ project and we were able to identify ways on how to improve more the FPOP The workshop is very useful for us since ideas pop up with different minds. In order to plan carefully for a more enhanced model in providing YFS
What is the most important thing you have learned? We must cooperate/ collaborate as a team That we still have a lot of areas in the advocacy to improve General programme of the entire SALIN+ areas – 4 chapters and
N.O. I learned about the experiences of our chapters. We can
duplicate their best practice which are replicable in our chapters and the institution in which I work full time as the Dean of the Nursing and Health Science Department I have learned that truly there is still a lot of work that has to be done We must always be youth friendly That we should have a proper planning framework for every project that we will be doing I value everything that I have learned and my role as a service provider in delivering YFS About the different projects proposed in our respective places involving youth The most important thing I have learned is that as we plan, we should make it sure that the plan is working progressively. I also learned to be more dedicated with my work as a volunteer Investing in young people can really help the society to change No harm in negotiating / trying The experience shared of other chapters on how they deal the weaknesses and strengths in SALIN+ The in-depth questions on how to evaluate a project Different strategies and our programme evaluation / standing The most important thing that I have learned today is the sustainability of the project Important thing that marked in my mind is to become a sustainable clinic – a continuous process of providing YFS to these targeted young people
How could the workshop have been improved? Workshop has been improved because now we can easily
identify data and goals we have met It’s just right for me
External evaluation of SALIN+ projects: Overview report 2010 81
I find it best for the purpose. I can’t think of a better way to do
As a young person being involved in an evaluation gives me the
it None Uhmmmm I can’t say anything about it because its my first time to attend a workshop on yours The workshop itself has improved the ideals and plans of the M.A. More interactive activities in between session we must inject icebreakers So far so good I suggest that an overall brief presentation of SALIN+ project was presented. Although everyone was very aware of the project, comparison of some of the accomplishments of the chapters can be good and evident I hope this could happen on a regular basis It can be improved by making the workshop more creative The workshop may have been improved if it is not only for one day
opportunity to learn how it is done. At the same time it gives me the opportunity to participate and with that it made me feel that my opinion is important and that I feel empowered. Thanks to Ms Susie Daniel for the untiring effort and patient in preparing the evaluation workshop I am so thankful that I was given the opportunity to share my experience in this activity – thank you Its great to review work done and assess what is still needed to be done I appreciate the way you approach us young people and indeed you’re a great evaluator! Not biased and you’re nice to talk with cause you are so open minded Do you need a caregiver? Nurse? Wanna apply …. Hehehe Thanks Ms Susie … it’s been a great experience that we can be able to be part of the whole evaluation process because we were able to visit other chapters and experience how they worked for the SALIN+ and measure whether we will adopt their strategy or improve it. We really appreciate how you score on the evaluation as it has a justice Thank you for being approachable. Thank you for the effort I learned a lot specially when it comes to family planning or reproductive health and how to help youth or young people I thank you for this opportunity – a BIG SMILE!! To IPPF and partners, thank you for the continuous support to young Filipinos, it’s a great experience working with you. I grow (not by height) by spirit in helping and providing services to the key populations. My exposure are blessing to do more and to share more to others. Looking forward to meeting you again! Thanks Bon voyage Sustenance and continuation of the project is recommended Job well done Ms Susie! No further comments…
Has the overall evaluation process been beneficial to you and your work? Yes Yes, it has Yes, very much Yes. I have actually learned a better way of evaluating a project Very beneficial. I will be guided of how to implement YFS
services better to the chapter Yes Yes Yes it is beneficial for future work – it can be used as a guide in
providing YFS Yes Yes Of course – yes. It helped me to evaluate myself as a peer
counsellor and for me to grow and develop more Yes – clarifications were made – I appreciate better the work of
young people Yes Yes it is beneficial to my job as government employee Yes Definitely yes The evaluation has a good impact for the volunteers and serves
as an eye-opener to strategize if the objectives of SALIN+ are met
Any other comments? Thank you Ms Susie and god bless
82 External evaluation of SALIN+ projects: Overview report 2010
Tunisia validation meeting evaluation findings
2
Time allocated for discussion
2
3
Overall assessment of the validation meeting
4
1
Completely
Fairly well
Approach
1
Were the findings and ratings accurate?
2
3
Will the evaluation framework and findings be helpful for your work?
3
2
Poor
2
Overall usefulness of discussions
Not at all
2
Fair
Good
3
Overall quality/relevance of presentations
Not very well
Very good
Overall ratings
Excellent
7 participants (2 young people left early – did not complete evaluations)
What was the most important thing you learned? What to put in place in order to perpetuate the project The ideas on the continuity of the project and the importance of
the monitoring and evaluation The importance of implementing a mechanism to insure the
maintenance of youth educators The evaluation gave us a clear idea of how to implement
projects in the future – what are the main issues that we need to take into consideration How to perpetuate the project
Any other comments about the meeting or about the external evaluation generally? The external evaluation would have been better if it had been
longer and took all the centres into consideration No It was too short (time wise)
External evaluation of SALIN+ projects: Overview report 2010 83
Annex 8 IPPF Terms of Reference for SALIN+ Evaluation The following pages are the original terms of reference developed by IPPF for the evaluation. Some processes were changed by negotiation during the duration of the evaluation.
Terms of Reference: Coordinating Consultant for the External Evaluation of SALIN+ Projects Background The Government of the Netherlands awarded extra funding to IPPF for 2009 and 2010 to strengthen the implementation of the Adolescent ‘A’ with a specific emphasis on increasing access for underserved young people and improving the quality of sexual and reproductive health services for young people. Funding was awarded to eighteen Member Associations (MA) – three per IPPF region – for a total of two years26. Additionally, each IPPF Regional Office (RO) was able to request funding for advocacy activities related to young people’s sexual rights and to buttress the ICPD + 15 campaigns. As part of the proposal development for the SALIN+ Fund, four global objectives were identified that align with both the IPPF Strategic Framework 2005-2015 and the 2007 Adolescent Review recommendations. Below is a brief summary of each objective and the corresponding activities. Member Associations were required to address the first two SALIN+ global objectives in their proposals, but strategic objectives 2 and 3 were optional. Each project selected its indicators from a pre-defined list of indicators, two of which were mandatory. The main objectives of the Fund included: To strengthen, increase and expand access to youth-
friendly sexual and reproductive health services for young people: this objective focuses on the expansion of services to the most underserved and vulnerable young people as well as ensuring quality, youth-friendly services. Member Associations are reaching out to young people who have never before received the services or information they need to make informed choices about their sexual and reproductive health, including young people living with HIV/AIDS, young sex workers, out-of-school young people and young LGBT. In doing so, MAs are promoting the sexual rights of all young people within their reach. MAs also are enhancing their capacity
to serve more clients by constructing new clinics, purchasing mobile clinics, and expanding their continuums of care. To increase institutional commitment to youth-friendly services and young people’s sexual rights: this objective aims to fulfil the right to participation of all young people as well as increase the overall commitment to their rights amongst MA service providers, staff and volunteers throughout the Federation. Indicators for this objective include an increase in the number of young people on governing bodies, adoption of child and youth protection policies and an increase in the proportion of young staff members in MAs. To promote the provision of comprehensive sexuality education and information to all young people: this objective promotes the accessibility of SRH information to young people with diverse needs, circumstances, and desires. It also advances a gender-sensitive, rights-based approach to sexuality education with a focus on peer education. To build supportive communities for the realisation of young people’s sexual rights: this objective aims to engage a range of stakeholders – from parents and teachers to religious leaders and politicians – who have an impact on the sexual and reproductive health of young people. MAs working with this objective are also encouraged to partner with organisations that specialise in youth programming.
Values and objectives of the evaluation The external evaluation will be based on and informed by the extensive Adolescent Review (2007)27, the IPPF Mid-term Review (2010) and MA SALIN+ project reports. Further, the evaluation ethics and approach must be in line with the sexual rights framework28 within which IPPF operates. Whilst the extent to which each MA applies sexual rights varies, it should serve as the benchmark against which we evaluate all of our work with young people. IPPF starts from the premise that young people are rights-holders that have the right to be actively involved in the decision-making process that impact not only on sexual and reproductive health and well-being, but also on other issues that affect their lives. This belief should be reflected in the evaluation, and a holistic view of the rights and development of young people will be adopted that includes, but moves beyond health outcomes and contraceptive uptake. The evaluation methodology should be empowering for IPPF Member Associations by allowing space for constructive, critical and collaborative thinking. The evaluation should help the MAs
26 Many Member Associations did not receive funding until mid-2009, and all MAs are permitted to spend the funds until 31 March 2011. Therefore, most projects are less than two years in duration. 27 To be provided by IPPF 28 To be provided by IPPF
84 External evaluation of SALIN+ projects: Overview report 2010
identify training needs and provide information or training on how to use the evaluation for future programme development. As an organisational learning tool, the evaluation should be should be accessible (not too technical) for all and speak specifically to IPPF’s work with adolescents and young people rather than being abstract. The eighteen SALIN+29 grant-receiving Member Associations are as diverse as their country contexts, but they are all piloting strategies to increase access for young people within their cultural, socioeconomic and religious environments. Although the evaluation will emphasize different themes in each country, the following are the general objectives: To evaluate the MAs’ project against the four global objectives
(see above) To identify and document promising practices in relation to
youth-friendly service provision and youth SRH programming that can be shared across IPPF and with external partner organisations To assess any unexpected added value or impact/results of the SALIN+ projects beyond the set goal and objectives of the projects
29 To be provided by IPPF
External evaluation of SALIN+ projects: Overview report 2010 85
Key Elements
Specific Questions
Means of verification
Objective 1: To strengthen, increase and expand access to youth-friendly sexual and reproductive health services for young people Health-based outcomes
Has the number of young people receiving SRH services increased during the
Rights-based outcomes
What has been the impact of youth participation in the design and
Focus group discussions with
monitoring of the project on young people? How has the project addressed gender inequity? How has the project increased access for young women? Do underserved groups (YPLHIV/AIDS, sex workers, OSYP, etc.) feel able to access services? Was it always like this? If not, what changed? Would young clients recommend the MA’s services to other young people?
project beneficiaries and staff Young client exit survey
project? If so, by what percentage? Has the MA been successful in reaching out to underserved young people? If so, how? Is there a wider range of services available to young people as a result of the project? How have the facilities of the MA’s clinical services improved as a result of the project? What impact is this having on access? How have the facilities of the MA’s non-clinical services improved as a result of the project? What impact is this having on access?
MA service statistics Project reports Staff interviews Youth-friendly check-list
Objective 2: To increase institutional commitment to youth-friendly services and young people’s sexual rights Institutional commitment
Has the project been effective in changing the values of service providers in
Interviews with staff and
relation to young people’s sexual rights? Has the project been effective in changing the values of MA staff members in relation to young people’s sexual rights? How has the MA implemented a child protection policy? What impact is this having on the MA in general and on service provision more specifically? How has the MA implemented a youth policy? What impact is this having on the MA in general and on service provision more specifically? Is there recognition within the MA of the barriers that young people face? Is there a commitment to addressing the barriers faced by young people going forward? Is the MA making an effort to understand the concerns, realities and SRH issues facing young people within its communities? If so, what are they doing? Has the MA made an effort to make known that it is a youth-friendly organisation through communication strategies aimed at attracting young people? How successful have these efforts been?
youth volunteers or peer educators General observations
Objective 3: To promote the provision of comprehensive sexuality education and information to all young people Comprehensive sexuality education
How is the MA promoting comprehensive sexuality education in schools or
other institutions that work with youth through the project? What successes can be identified? In what other ways did the project promote the provision of CSE to young people? Are young people more aware of their sexual rights as a result of the CSE promotion?
Interviews with project staff,
external stakeholders and young beneficiaries
86 External evaluation of SALIN+ projects: Overview report 2010
Information, education and communication
Are the messages communicated by the MA in-line with the IPPF 4IEC
Analysis of IEC materials
checklist? Are peer educators equipped with enough SRH knowledge to conduct outreach with their peers? Are peer educators trained and confident in communication strategies for conducting outreach with their peers? What have been the greatest challenges in the MA’s peer education programme?
distributed by the MA (including radio adverts, pamphlets, web pages, etc.) Interviews with peer educators and staff
Objective 4: To build supportive communities for the realisation of young people’s sexual rights Advocacy
Has the MA advocated for changes to laws or policies that limit young
people’s access to SRH services or information? If so, what strategies were used? What successes can be identified? Does the MA see a role for itself in advocating for the above changes? If not, why not? In relation to young people’s sexual rights, is the MA an organisation that a) exerts influence; b) resists social norms/practices/laws that act as barriers; OR c) acts as an agent of change? Involvement of parents and other gatekeepers Community awareness
General observations Interviews with staff and
external stakeholders
How successful has the project been in involving parents? Other gatekeepers?
Interviews with staff and
What strategies were used? What impact has the involvement of parents and other gatekeepers had on young people’s access to SRH services and information?
Focus group discussion with
How has the project addressed cultural, social and religious barriers? How
successful have these interventions been? Are the communities where the project has been operating more conducive environments for young people to realise their sexual rights than those where the project is not operating?
young beneficiaries parents or other gatekeepers General observations Interviews with young
people and staff
Cross-cutting themes Sustainability and cost effectiveness
Promising practices
Which project interventions and/or activities have been incorporated into the
MA’s annual work plan for 2011? Has the MA made an effort from the beginning of the project to conduct activities in the most cost-efficient way? What challenges will the MA face in sustaining essential project activities without further restricted funding? Will the MA have to cut staff after the SALIN funding ends? If so, what impact will this have on services for young people? What intervention(s) have been most successful in increasing the number of
young people accessing services and information? Explain in detail. What have been the most successful strategies for reaching out to underserved young people? What three-promising/good practices would the MA like to share from its experience under SALIN+? Added value
Are there any results from the project that were unexpected? If so, provide an
explanation of what they are and what their impact was/is. How is the MA contributing to the strengthening of the national health systems and youth-friendly service provision in the public sector? How has project impacted on the MAs’ data collection mechanisms?
Analysis of financial and
programmatic reports General observations Staff interviews
MA progress reports Staff interviews Focus group discussions with
young people
MA financial reports General observations External stakeholder
interviews
External evaluation of SALIN+ projects: Overview report 2010 87
Methodology Both quantitative and qualitative methods should be used for the in-country evaluations. The evaluation team need to consult with primary and secondary stakeholders and with recipients of the services. Young people should be involved in interviewing peers and other stakeholders. As per the above table, examples of research methods include: Service statistics: use of service data provided by IPPF to give
an indication of the trends in service uptake by young people over the baseline and project periods Data collection mechanisms: examination of the data collection mechanisms in-country to provide insight into the challenges faced by the MAs in collecting, presenting and utilising data for project improvement Group discussions: focus group discussions with peer educators, young people and other primary and secondary stakeholders (both male and female) on the implementation and results of the project, quality of services, client satisfaction, youth participation, etc. Key informant interviews: key informant interviews with external stakeholders, project staff and health service providers involved in the project on the implementation of the project and its results with regard to the project’s objectives, success factors/ challenges, the MA’s institutional capacity, partnerships, community participation, etc. Participatory research method: the use of participatory research methods and in-depth interviews with young people, in order to determine the impact the project has had on their lives. Site observations: Observations using an agreed checklist on youth-friendly service provision
Duties and responsibilities of coordinating consultant We envisage that the development and co-ordination of the evaluation as well as the final report will be led by one main consultant or research institute. The coordinating consultant has the option of utilising local consultants to do the in-country evaluations, if the consistency and quality of the evaluation can be assured. The coordinating consultant will be responsible for the following: 1 Development of an overall evaluation framework and template for in-country evaluation reports for approval by IPPF 2 Analysis of IPPF service statistics, project data and global indicators for the 18 SALIN+ grant-receiving Member Associations
3 Analysis of Department of Health data in relation to adolescent (under-25s) sexual and reproductive health in each of the 3 – 5 countries participating in the external evaluation (to give context to the evaluation in each country) 4 Preparation of a guidance document and instructions for incountry evaluations to be used by local consultants to ensure consistency (if applicable) 5 Pilot of the evaluation framework in one SALIN+ MA and make necessary adjustments before sending to in-country consultants 6 Coordination, collate and ensure the quality of in-country reports from local consultants 7 Prepare a final evaluation report that incorporates the 3-5 incountry reports; the final report should include: zz Executive summary detailing the key findings, conclusion and most important recommendations for IPPF and its MAs zz Key achievements and outcomes zz Progress in relation to each of the four global SALIN+ objectives zz Explanation of the unintended results and added value of the project in relation to national health system strengthening zz Challenges and barriers that still exist in relation to the realisation of young people’s sexual rights zz Promising and good practices in relation to youth SRH programming zz Assessment of effectiveness and sustainability of the funding zz Recommendations for follow-up activities and improvement zz Advice for MAs on how to utilise the evaluation report to inform programme development in the future 8 Deliverables: The main deliverables expected of the coordinating consultant are as follows: zz Development of a draft framework and methodology for incountry evaluation of SALIN+ projects zz Analysis of service statistics from all 18 SALIN+ MAs zz Pilot of the draft framework in one SALIN+ MA zz Adaptation and adjustment of the draft framework following the pilot; production of a final framework zz Full evaluation in 3-5 SALIN+ MAs (countries TBC) zz Production of between 3-5 country reports (including the pilot country) zz Production of a final evaluation report that incorporates the findings from all country reports The time line of the evaluation will be from January until April 2011, with a final report due by 1st June 2011.
Profile coordinating evaluator/ research institute IPPF will recruit one coordinating consultant or research institute to oversee the evaluation between January and June 2011. The required skills and experiences are as follows:
88 External evaluation of SALIN+ projects: Overview report 2010
Extensive knowledge about youth sexuality, sexual rights and
gender Commitment to the sexual rights of young people, including
an acceptance that all young people are sexual beings and that some are sexually active Solid understanding of the essential components of youthfriendly service provision Previous experience in monitoring and evaluating SRHR projects using participatory evaluation methodologies Interviewing skills, especially with young people Committed to youth participation Able to administer complex evaluations Sensitivity to different cultural and religious contexts Capacity to analyse data in a systematic way Ability to present complex information in an accessible way Excellent writing, analytic and communication skills Fluency in written and spoken English Ability and commitment to deliver the expected results in a short period of time
External evaluation of SALIN+ projects: Overview report 2010 89
Photo credits
Page 08: IPPF/Susie Daniel, Guatemala – 2011 Page 09: IPPF/Susie Daniel, Mozambique – 2011 Page 12: IPPF/Susie Daniel, Mozambique – 2011 Page 13: IPPF/Susie Daniel, The Philippines – 2011