What’s the Goal? What’s the Purpose? Observations on Human Rights Impact Assessment Bård Anders Andreassen and Hans-Otto Sano
Norwegian Centre for Human Rights University of Oslo P.O Box 6706, St. Olavs plass NO-0130 Oslo Norway Email: peris.jones@nchr.uio.no Tel. +47-22842020 Fax +47-22842002
Research Notes 02/2004
Copyright: The Norwegian Centre for Human Rights / Bård Anders Andreassen and Hans-Otto Sano 2004 The manuscript forms part of the research programme connected to the Centre’s South Africa Programme Research Notes is a series issued by the Norwegian Centre for Human Rights Editor: Bård Anders Andreassen Technical consultant: Christian Boe Astrup Research Notes is available on the Centre’s website http://www.humanrights.uio.no/forskning/publ/publikasjonsliste.html Any views expressed in this document are those of the author’s, and do not necessarily represent those of the Centre. ISSN: 1503–1349 ISBN: 82-90851–64-2
CONTENTS INTRODUCTION ........................................................................................................................................ 5 ANALYTICAL CHALLENGES OF HUMAN RIGHTS IMPACT ASSESSMENT............................ 6 THE PURPOSE OF HUMAN RIGHTS IMPACT ASSESSMENT ....................................................... 8 LIMITATIONS .......................................................................................................................................... 10 DEFINING GOALS AND OBJECTIVES ............................................................................................... 11 THE PROBLEM OF ATTRIBUTION..................................................................................................... 12 INDICATORS AND THE PROJECT LOGIC........................................................................................ 12 CONCLUDING REMARKS ..................................................................................................................... 23 REFERENCES ........................................................................................................................................... 24
Research Notes 02/2004
INTRODUCTION1 This paper addresses the use of indicators in assessing the impact of human rights projects in fulfilling their objectives. The terms “human rights projects” refer to development initiatives defined and designed to enhance human rights in societal contexts, and conducted by public or non-governmental organisations and actors. The paper highlights the need for formulating indicators that are accurate and appropriately related to the goals and objectives of human rights projects. It assumes that this has been neglected in international human rights project work, as well as in human rights research. Why is it important to pay attention to indicators, goals, objectives and monitoring of human rights projects? Three reasons seem particularly important. First, the construction of accurate indicators is clearly needed in order to establish the possible impact of and obstacles to the success of human rights projects. This need for better analytical tools and frameworks in this field of human rights advocacy is being voiced among national and international human rights organisations, as well as among multi-lateral agencies and bilateral donors. Second and closely related to the first reason, an important objective of human rights projects is to facilitate changes in institutional, legal and social rules, procedures and behaviour. However, due to the lack of appropriate descriptions of project goals and objectives it is often difficult to document such changes. Therefore, there is a need for identifying indicators that can be used effectively to document changes, or, put differently, measure and identify changes based on knowledge about options and obstacles to the implementation of projects in social, cultural, economic and political contexts. Third, a systematic application of indicators will strengthen transparency and accountability in the institutional implementation of human rights. Such projects have become a “professional activity” in democracies as well as in authoritarian states, and the growing use of human and financial resources has to be “accounted for” in order to nurture future moral and material support. But is it at all possible to measure the impact of improvement or deterioration in human rights conditions of what is often small projects of short duration? Some argue that the time horizon of “the object of change” – the prevailing human rights conditions - is so elusive that measurement of impact and result becomes unfeasible and unrealistic. It is also often argued that the use of resources in support of human rights has to tolerate a high degree of risk and potential failure. It is a “risk” activity. 1
This paper was prepared for NORAD’s Technical Division, Oslo, and submitted in June 2003. The paper summarizes lessons learnt during two workshops jointly organised in January 2003 by NORAD and the two Pakistani human rights organisations, Society for the Protection of the Rights of the Child (SPARC) and Human Rights Commission of Pakistan (HRCP). The discussions during these two workshops helped to draw conclusions and formulate recommendations presented in the paper.
5
Observations on Human Rights Impact Assessment
In this paper, we argue that even if there are many methodological problems connected to making accurate impact analysis in this field, it is still possible to develop better methods and analytical tools for making inferences about the impact of human rights projects than has been done so far. Over the last decade, support for international human rights and democratisation has grown steadily as an area of international development co-operation, and since the mid-1980s, there has been a shift in policy orientation from principle commitment to actual support for human rights programs and projects. Comparatively large resources have been used on supporting human rights and democracy interventions (projects and programs) by governmental, inter-governmental and non-governmental institutions and organisations. These developments have created a need for developing indicators and approaches for measuring and assessing impact of human rights and democracy projects. At the international level, the interest among human rights organisations at local and national levels is matched by international organisations to learn more about the effects of joint (or “partnership”) activities, requiring the construction of analytical models and tool for impact assessment. This field of analysis is interrelated, and to some extent overlapping with a trend towards measuring democratic performances in “new” and old democracies (cf. such terms as “democracy audits”, “democracy assessment”, “good governance measurement”.) In order to learn more about practices, possibilities and limitations of identifying indicators, and to use them in assessing human rights projects, the authors of this paper took part in two workshops organised jointly by the Norwegian Agency for Development Cooperation (NORAD) and two Pakistani human rights organisations in Pakistan in January 2003. These organisations were the Society for the Protection of the Rights of the Child (SPARC) and the Human Rights Commission of Pakistan (HRCP). The workshops offered opportunities for discussing the indicators, the issue of measurement and impact assessment based on first hand experiences of these organisations.2 This research builds on these experiences. The objective of the paper is to draw on the experiences of the workshops, and not assessing these two organisations. More generally we use the insights from these endeavours to contribute to constructing better tools for human rights impact assessment.
ANALYTICAL CHALLENGES OF HUMAN RIGHTS IMPACT ASSESSMENT We shall cluster our discussion around four analytical challenges. These are the matching between the objective, activities and outcome of a project; the identification of type of change that follows from a project; the issue of aggregation of impact, and the task of
2
Reviews of these workshops (written by the authors of this paper) were submitted to NORAD, but have not been included here. 6
Research Notes 02/2004
identifying key result indicators.3 Below follows a brief discussion of each of these challenges. Matching Human rights projects are usually carried out through a number of activities. A common shortcoming, however, is that projects too often are implemented and assessed as if the activities were the purposes and goal of the project, for instance legal advocacy, training human rights legislation, etc. However, training is not an objective in its own right, but an activity that pursues specific objectives and ultimately - an overarching human rights goal. What has to be accounted for in the last instance is not the performance of activities but the impact of these activities in bringing about human rights change. Therefore, projects should define rights-related objectives and at an overall level the goal that go beyond the particular activity (or activities) such as training, dissemination, and other types of advocacy work. Matching of activity and objective would imply, for instance, establishing the specific level of empowerment that the trainees should possess in order to have the capability to make human rights claims. Types of change In general terms, human rights interventions address three types of changes. First, changes in the conduct of state policies and law enforcement. In this case, human rights assessment requires indicators of conduct and operation of state authorities’ capacity to facilitate and ensure rights-effective economic, social, and political changes. Second, human rights interventions may facilitate social change conducive to human rights norms, for instance, non-discrimination between men and women. Indicators of social change are indicators of result because they relate to actual enjoyment of rights. Third, human rights interventions may improve institutional capacities through capacity building. Over the last decade, the strengthening of civil society organisations has been stressed as an important domain of human rights promotion. In this case, indicators should reflect the capacity of civil society to monitor human rights or to network and campaign in order to create advocacy directed towards the state. This requires indicators of institutional change. Aggregation of impact The chief purpose of human rights assessment is to examine how a project influences the overall human rights situation in a local or national community. The assumption that a single project contributes to the overall (“macro”) human rights situation is, of course, benign but often very hard to document. An assessment study, nevertheless, should address this issue, and at least discuss the limitations of making realistic assessments of the aggregation of impact from the project level to macro level.
3
Human rights projects are also referred to as human rights “interventions”. By “intervention” we refer to the introduction of projects with human rights goals into social contexts. Human rights projects often have the character of being introduced by non-governmental organizations, backed by the financial (and other) support of international partners (including aid partners), or by governmental institutions. They are, in other words, “acts of intercedition” in order to support or strengthen the rights based power and capability of those on whose behalf the project is undertaken. The term “intervention” stresses this nature of human rights projects. 7
Observations on Human Rights Impact Assessment
Key result indicators Aggregation of impact requires the construction of indicators that can help to interrelate and attribute changes at the micro level - stemming from successful projects - to changes in human rights conditions at national (macro) level. Below we point out that human rights organisations often operate in unstable and volatile political contexts of rapid change (political transition), and that a main wish is to contribute to these changes. At the same time, the human rights contents or dimensions of the work of organisations in such contexts are often unspecified as organisations formulate their activities in terms of contributing to “a process of change”, “democratic transition”, “good governance”, or other broad social or political goals. In order to assess the impact of human rights projects in such contexts, however, the goals need formulation in human rights terms, and are referred to as key result indicators. In section 5, four categories of key result indicators are suggested; indicators of incorporation of human rights law, indicators of compliance with political rights, indicators of compliance with civil rights and indicators assessing economic and social rights. This paper follows up experiences with NORAD’s Handbook in Human Rights Assessment (Oslo: NORAD, 2001). The Handbook is mainly a tool for assessing potential, planned and/or likely positive or negative effects of a programme or project under review for potential approval of funding. Its main use is assessing project documents in the pre-appraisal, appraisal and appropriation phases of the programme cycle. The present paper addresses some methodological hurdles and requirements for making contextual impact assessments of human rights project/programme that have been implemented, and stresses the need for developing reliable indicators for documenting and measuring positive and negative impact of human rights projects and programs.
THE PURPOSE OF HUMAN RIGHTS IMPACT ASSESSMENT The UNDP Human Development Report 2000, covering human rights and human development issues, refers to the construction of reliable indicators for human rights impact as “a cutting edge area of [human rights] advocacy” (p. 89). The report refers mainly to the use of statistical indicators that address the issues at a macro level, and suggests that indicators are tools for making better policies and improving monitoring. Indicators are also means to identify unintended impacts of laws, policies and practices, for addressing how different actors influence the realisation of human rights and for revealing whether the obligations of these actors are being met. Indicators are also needed in order to give early warning of potential violations of human rights, and suggest preventive action. These various functions of indicators highlight that an important objective of human rights impact assessment is to identify and measure accountability of actors of human rights advocacy (not just the state), and to try and document short- and long-term impact on specific human rights, or clusters of rights (for instance the rule of law). In discussing human rights impact of projects and policies we distinguish between two dimensions of
8
Research Notes 02/2004
state human rights obligations, that is, obligations of conduct, and obligations of result. According to the obligation of conduct (the process variable), indicators must reflect or document the willingness of the state to comply with human rights treaty obligations over time. According to the result variable assessments should be made about capacity and success of bringing about the goods, values, immunities etc. set out by individual or group human rights. Indicators are tools for verifying positive impact, or documenting failures to achieve expected impact. At project or program levels, indicators can express a quantitative relation, using statistical tools, or alternatively being used for a contextual, qualitative assessment of potential (planned) and/or actual impact of a project on human rights enhancement. To take an illustration: A quantifiable measurement of torture is the number of torture victims used to measure the impact of a campaign against torture. The challenge here is to obtain reliable data as regards changes in the number of victims.4 A qualitative indicator may be the perception among inmates of a change in the behaviour of prison guards, documented through a sample of interviews. Due to the political sensitivity of documenting human rights violations in many country situations, the qualitative information will be relevant and realistic in a number of cases. Indicators for assessment and measurement of human rights interventions can be identified at various stages of a project cycle. In this paper we apply the logical framework approach as a design methodology that structures the identification of indicators at various points and dimensions of a project cycle.5 Although the indicators discussed below largely apply to human rights interventions of non-governmental organisations, they are equally relevant to public (or state) projects and policies. Human rights indicators can also be used to document and verify positive or negative impact on the upholding of rights of individuals or groups. In spite of good intentions, human rights projects may have unintended, or implicit negative effects to various degrees. Human rights projects are inherently interrelated with other development practices and interventions. While development aid aims to eradicate poverty, and redress inequalities and injustices depriving people of their capabilities to lead a decent life, human rights are expressing these issues as universal principles and entitlements that all people ought to have anywhere. Human rights can be seen as meta-norms of development and development policies (including trade, investment, growth etc) Development policy even in terms of trade or macro-economic policy may have human rights implications. Mainstreaming human rights in the development context implies that the human rights risks and implications of development programmes should be assessed. Indicators measuring risks and sensitivities are crucial in this context. Human rights indicators document and verify human rights-based development, but the assessment – the
4
This could draw on the Basic Principles for the Treatment of Prisoners, UN resolution 45/111 adopted by the General Assembly on 14 December 1990. 5 For a discussion of the logical framework approach, se for instance Sida 2000. 9
Observations on Human Rights Impact Assessment
interpretation of the values of the indicators - should be made in manners that are sensitive to the societal context. We assume that when development funds are being used for supporting human rights organisations – the subject of this paper – the performances of these institutions should be assessed and evaluated by the same methodological principles as other development programs. The logical framework approach offers a widely used method in this regard (as further discussed below).6
LIMITATIONS Assessment of human rights impact documents the scale of impact, that is, whether it has been marginal, significant or something in-between. This is important information for assessing the rates of return in relation to resources utilised, and also in determining future priorities. As noted, and returned to below, two of the most difficult issues in measuring human rights impact, are to ensure that a documented effect is attributable to a particular intervention and to quantify impact by scales or other measurement. In measuring goal achievement and fulfilment of objectives, human rights NGOs face the challenge of documenting the relationship between human rights activities and change in terms of human rights improvements that follow from these activities. This is the problem of attribution referred to above. Dealing with the problem of attribution requires monitoring and reviews. This could be either through monitoring undertaken by NGO organisations themselves, or through external evaluations. ------------In the remaining part of the paper, we discuss the development of indicators under three general headings: First, we highlight the need for being specific about the goals and objectives of human rights projects. If goals and objectives are unclear and not properly defined, we do not know what to measure, and identifying indicators becomes a futile exercise. Second, we discuss the problem of attribution in further detail. Third, we discuss the relationship between indicators and “the logical framework” of the project, and introduce a case of illustration for identifying an assessment structure of an individual project. This discussion includes comments on the added value of human rights projects to overall goals of human rights promotion, and relates three types of change that human rights aim to achieve to possible indicators for assessing such changes. We further emphasise the importance of conduct monitoring of projects, as to the effects this may have on outcome of the project, as well as the implementation and management of the project. In a subsection we distinguish between first order and second 6
We have observed a good deal of reluctance to the feasibility of the log-frame approach in assessing complex processes of institutional change, and creation of awareness (as is often the case in human rights projects). This reluctance reflects position of critical studies of the log-frame approach, well known from the development literature. A suggested alternative is to work with development pathways, that is, narrative descriptions of the project logic (without strictly defining goals, purposes, outputs etc). However, it is not at all self-evident that this alternative escapes the need for making assessment of results, and hence the identification of indicators for documenting and verifying impact. 10
Research Notes 02/2004
order impact, reflecting the relationship between the individual and institutional levels of impact, and comment on human rights indicators in situations of political transition. For some of these subjects we identify a set of Guidelines for construction and application of human rights indicators at the project level. These Guidelines (and further elaborations) may help human rights organisations and institutions to pay more focussed attention to the possibilities for measuring impact and at the same time be aware of constraints and limitations.
DEFINING GOALS AND OBJECTIVES Goals and objectives of projects and policies describe the outcome of process of change, usually resulting from a planned intervention. Goals and objectives represent desired situations and conditions and are not description of activities. Definitions of the goal(s) of a project require examination of • the coherence of the interrelationship between the goal and the objectives; • whether the goal and the objectives express a perspective of change; and • whether the objectives can be achieved within the project period. The goals and objectives of a project determine the choice of indicators. This is often a weak point of project documents. Too often, goals and objectives are not operationalised or realistically defined. It is also often neglected that goals are long-term accomplishments, while objectives are ambitions and achievements in the course of the specific project cycle. An objective shall significantly influence on the goal. When the distinction between goals and objectives is unclear, it is hard to evaluate a project at all. Strangely, human rights organisations often do not define goals and objectives in human rights terms. One reason for this may be a limited knowledge or competence in human rights, and hence, in formulating goals and objectives in human rights terminology. Or it may simply result from a lack of attention to make appropriate objective and goal formulations. Organisations often (and for good reasons!) focus on strengthening their institutional capacities and creating awareness of human rights through advocacy activities. Hence, when indicators for documenting the impact of their work are set out, organisations in such environments usually refer to institutional performance, such as number of workshops or production and dissemination of human rights information, while neglecting the long-term human rights goal. In brief, therefore, in order to verify human rights impact it is important to define the overall goal as well as more specific objectives in rights terms. The objectives and the activities that lead to them are justified essentially by the contribution they make to the overall goal of strengthening and promoting specific human rights.
11
Observations on Human Rights Impact Assessment
THE PROBLEM OF ATTRIBUTION The problem of attribution has two dimensions. First, a change in human rights practices may stem from various sources. For instance, legal reform may stem from the work of a large number of human rights organisations, lobbying, change in political leadership, etc. The problem is to determine the impact of each of these factors of interventions, and the relative effects they might have had on changes in human rights practices and fulfilment. The difficulty of making exact attributions implies that we often have to accept assumptions about likely impact of a project. It’s important to keep in mind that decisions about support to projects or policies are made under uncertainty, calculating with “risk factors�. Risks analysis, moreover, has to be based on knowledge about political context, social conditions and structures, cultural tradition, degree of institutionalisation and legal conditions. It may be equally difficult to document the contribution of a particular source of support for a human rights project. Human rights institutions and organisations often have many partners supplying them with funds or other resources. The problem of determining the net effect of the support from one source might be overcome by introducing very detailed regulations of accounting about the use of particular grants, or by so-called formative process research. However, these are often impractical measures that are not feasible for one reason or another (for instance, they may be too expensive), even if this reduces the accuracy of the assessment. The time factor may also limit the reliability (and accuracy) of the assessment. A human rights project can have little effects in the short-term, although it may have important long-term impact. Awareness-raising campaigns that give little immediate gains may have positive effects in the long-run. At the same time, indicators documenting shortterm impact may not tell us much about the sustainability of a project. Accordingly, the problem of sustainability may not be seen as pressing in the short-term, but critical in a longer time perspective. Again, assumptions about sustainability require knowledge and insights about political, socio-cultural, legal and institutional contexts. Human rights projects may target changing institutional cultures, elite attitudes or the competence of common people, all of them practices that usually are changing very slowly. Human rights changes are therefore often long-term and gradual, while many human rights projects have short time horizons. This represents a dilemma, which requires a long-term analytical perspective with time series data, and follow up studies long after the conclusion of the project. Again, the impracticality and the expenses involved may require a relative high threshold of uncertainty as to the human rights payoff a project.
INDICATORS AND THE PROJECT LOGIC In order to demonstrate the identification of indicators at each dimension of the logic (or structure) of the project, Table 1 presents a project aiming at the improvement of a human rights dialogue between the state and civil society. The case applies the log-
12
Research Notes 02/2004
frame approach as the organising structure for project design and implementation. The table also illustrates that any indicator must have a means of verification, that is, a reliable source of information and documentation to draw on. Referring to the table, a project design requires that the intervention logic is systematic, meaning that the movement from the activities defined in the bottom to the goal defined in the top of the table is coherent. The “logic� is the following: Under the preconditions given in the bottom right column at level (1), inputs should be sufficient to perform the activities of the project, and produce the output planned at level (2). The objectives of the project will be produced under the assumptions at level (2). Finally, if successful, the objectives will, under the assumptions at level (3) contribute to the implementation of the goal of the project. Some further observations on the case presented in the table may be of interest. First, as the project seeks to influence the willingness and capacity of the state to dialogue with civics, the project has a long-term goal. The goal is to achieve changes of institutional cooperation and behaviour, and not, for instance awareness-raising among state officials in itself. The logic of the project is to combine documentation, training and lobbying in order to nurture behavioural changes among state officials and make them enter into dialogue with civic society (the objective). Such dialogue, as suggested, would help to promote respect and fulfilment of human rights obligations of the state (the goal). Lobbying and pressure do not become goals in themselves, but means to create institutional change, and facilitate dialogue. Second, while indicators of the goal are rights-based (aiming to secure state commitment to reduce the number of violations of civil and political rights), this goal is achieved through institutional changes in attitude and behaviour, reflecting a commitment to implement treaty-bound human rights obligations, and documentation of this in reports according to their treaty obligations.
13
Observations on Human Rights Impact Assessment
Table 1 Case for illustration: Enhancing Human Rights Dialogue Goal Indicators Means of -The state report Enhancing respect for verification civil and political presented to the UN in - Observations to rights through statetime the State report of civil society dialogue - The state report the UN Human on reporting to the indicates a Rights Committee UN commitment to deal with HR violations Objectives Indicators: Means of - The capacity of - State preparation verification state officials to of reports to the - Procedures of report on progress on committee improved preparing report in civil and political - Quality of state Ministry x rights to the UN reporting improved - Assessment by Human Rights - Quality of NGO UN HR Committee Committee is shadow reporting of the State Report improved improved - The NGO shadow - The capacity of report is used by the NGOs to shadow HR Committee report to the Human Rights Committee is improved Outputs Indicators Means of - Report on the - X no. of reports verification “State of HR� distributed - The Manual and produced - X no. of the training reports - Manual of HR participants from state - Evaluation reporting produced. ministries in x courses reports from training -Joined training - Course evaluated sessions successfully positively by NGO as - Project progress implemented in x well as by state reports localities participants Activities - Preparation of the Annual State of HR in country - Preparation of training materials on HR and on HR reporting to UN based on cases from Annual Report
Inputs - Budget - Staff - Meeting space
14
Assumption (3) The ministry heads ready to change procedures of reporting
Assumption (2) Key ministry heads allow relevant staff joint training with NGOs
Precondition (1) - A commitment to engage in cooperation on HR issues among key state and non-state actors (incl. NGOs)
Research Notes 02/2004
Third, the objectives are accomplished through the twin activities of joined training and documentation. The objectives (capacity development), are conditions for goalachievement. As noted, too often projects define the goal as well as objectives as activities, for instance lobbying and pressure, neglecting what the outcome of process is expected to be (the goal). In some cases, organisations define so many objectives for one project that they are reduced to activities just because no one has the time to undertake a more focussed approach. Fourth, an output, for instance an Annual Report on the State of Human Rights, is not an objective in itself, but a means used in human rights advocacy, lobbying or training. And fifth, a precondition for improving the dialogue between the state and civil society organisations (the goal) is that the state is willing to communicate and ready to accept the establishment of a forum or other vehicles for dialogue. Without the existence of this prerequisite (political will), there is little use in planning and implementing the project. General Guidelines on the relationship between Goal, Objectives and Human Rights Indicators •
There should be only one goal for any one project, and just two or three objectives. Beyond that, objectives tend to be confused with activities. Organisations are accountable to the fulfilment of the goal and the objectives that they have defined.
•
While the realisation of the long-term goal can be seen as a process, the objectives have to be realised in the course of the project, and the individual project is but one source of contribution.
•
Documentation, lobbying and training are recurrent and important activities of human rights projects. However, these are activities and not objectives in themselves. It is, at any rate, important to be explicit about the sort of conditions (objectives) to which activities should lead. Indicators for Added Value
Indicators are quantitative or qualitative statements that can be used to describe human rights in situations and contexts and to measure changes or trends over a period of time. They are pieces of information that may provide insight into matters of larger significance, i.e., they may be seen as small windows that provide a glimpse of a wider landscape. In designing the project, it should be assured that indicators are verifiable, that they are specific and relevant, and that they are rights based. Usually, human rights organisations have no difficulties in defining indicators of the outputs they intend to produce, such as training, analytical reports, pamphlets, legal advice, lobbying etc. Activities are usually verified by using indicators like number of trained persons, reports or books produced, clients served, and meetings with people who may influence political decisions. These are indicators of activities.
15
Observations on Human Rights Impact Assessment
Moreover, it is often more challenging to construct human rights indicators that measure immediate objectives and long-term project goals. When indicators have been identified at the output level (activities), the next step is to move from a description of quantitatively measured output to analysing how this output contribute to the project objectives. Objectives can include enhanced awareness, change of attitudes, change in institutional or legal behaviour or social change. And as noted, the ultimate purpose of human rights impact assessment is to indicate how these objectives contribute to the goal of human rights improvements on the ground. In moving up the scale from output (from activities) to objectives and goal, focus is on the added value of the project to human rights promotion and protection. Too often, organisations are vague and imprecise when addressing (if they address) a project’s contribution to change in human rights conditions. Types of Change and Related Indicators Put simply, human rights projects may create three types of human rights changes. The first type is state reform and implementation of policies such as administration of law, enforcement of law, legal reform etc. This type concerns the conduct of states as referred to, and their obligations to respect, protect and fulfil human rights. It requires that political elites and the state apparatuses have a political will to adhere to human rights principles. One important indicator of political will is the ratification of international human rights treaties. Second, interventions may facilitate social changes conducive to human rights norms. Social prerequisites inherent in human rights can be policies addressing social inequities, gender discrimination, land reforms to help landless people, health policies etc. Indicators describing such changes are indicators of result because they relate to actual enjoyment of rights. Third, civil society organisations often have important functions in human rights promotion. Their effectiveness, however, relies on a variety of factors, for instance their capacity to monitor human rights or to network and campaign for human rights promotion, and lobby for the state to undertake its treaty obligations. Indicators would typically be indicators of competence and capacity building, with the objective of enhancing institutional capacity Examples of indicators of the conduct of states are - indicators measuring changes in specific human rights violations, for instance reduced rates of torture; - indicators assessing state commitment through legal, institutional or procedural changes, for instance introduction of codes of conduct in public offices, telephone complaint lines, instigation of public investigator offices in various fields, etc.; - indicators measuring state commitment through resource allocation, for instance increases in budget allocations to primary education.
16
Research Notes 02/2004
Examples of indicators of result are - changes in gender discrimination, e.g., the changing ratio of girls to boys in schools; - changes in recorded minority or indigenous group discrimination; - measurement of access to resources such as access to health facilities, education and housing; - measurement of standard of living in terms of poverty rates, income levels etc.; - measurement of access to justice. Examples of indicators of capacity building are - indicators that measure civil society capacity to promote and implement rights; - indicators of civil society capacity as watch-dogs, and capability to hold dialogue with the state; - indicators of civil society advocacy and capacity to pursue litigation; - indicators of civil society networking capacities. General Guidelines on the Construction of Human Rights Indicators • General observation: Human rights projects should include indicators that refer to changes in terms of respect, protection, and fulfilment of rights. Democracy projects may refer to human rights, but often in an unspecific way. If human rights change is part of the indicators of democratisation, the process of change identified should be more specific than referring to the strengthening of human rights, for instance identifying the right or cluster of rights that they address in the democratic context, i.e., the specific measures of political participation, the release of political prisoners, or the freedom of expression of the media. . • Quantitative or qualitative measurement? Whereas indicators of output are often defined in quantitative terms, indicators of goals and objectives may be require qualitative analysis and contextual interpretation. In some cases, a growing state commitment can be measurable in quantitative terms, for instance, lower rates of malnutrition reflecting a change of agrarian and land policy. However, not all state conduct is possible to convert into quantitative indicators, and it may be utterly expensive and time-consuming to establish quantitative and reliable indicators, and collect data. The cost-benefit dividend may imply lower demands for accuracy in certain situations, for instance in situations of high levels of political instability. • General requirements to indicators: Indicators should be specific, measurable, acceptable, relevant, and time-specific (cf. the abbreviation SMART). Indicators used in project assessment should always be verifiable, which requires that reliable documentation is available or collected and compiled, specific in terms of measuring or describing the achievement of a particular right, and relevant for the right/rights described, and the socio-cultural and political context of the implementation of the right. For instance, redressing unemployment and lack of income in highly industrialised society by distribution of land for agriculture is not a realistic option, while this may be exactly what is needed in agrarian societies.
17
Observations on Human Rights Impact Assessment
The fulfilment of rights can generally be measured in quantitative terms, at least at a nominal level of “either”/”or” (fulfilled/not fulfilled). However, usually we want to have more information about the degree of fulfilment, which due the complexity of building exact indicators and to collect sufficient data, cannot realistically be compiled. We therefore have to use proximate assessments, and indicate trends that rely on qualitative assessments of a particular right contextually interpreted. • The process of defining indicators. Indicators are often defined “in last minute” when a project document is formulated. It is important, however, to take the identification of indicators seriously, and to construct indicators for each step in a project cycle. To communicate the choice of indicators to partners and to consult with them on the choices made may enhance trust and transparency among partners, and make it easier to have needed adjustments of objectives accepted by partners (or monitoring bodies) during a project cycle. • Key result indicators and benchmarks: Indicators of results measure long-term accomplishments. For instance, indicators defined to measure the UN Millennium Development Goals assess the outcome of development processes that have been agreed upon by the international community (for instance reducing poverty by 50 % between 2000 and 2015). Key result indicators communicate important national or international goals. Small non-governmental organisations should be careful in defining key result indicators because they do rarely have the resources needed to achieve them, for instance to secure free and fair elections, or ensure free primary education for all. But for an international partner institution, acting in tandem with other international institutions and with governments with long-term human rights strategies, key result indicators are means to focus their interventions. Indicators and Monitoring Monitoring is a continuous or periodic supervision of the progress of a project or policy. In planning a project, a framework for monitoring progress should be defined as part of the project document, clarifying who is responsible for monitoring, and specifying the indicators to be monitored. Monitoring of a project is an instrument that allows management to govern and control the process of implementation and to report about it to stakeholders. Monitoring, therefore, has two general purposes. First, it is an input to management for comparing and assessing plans and actual implementation, and triggers analytical reflection and questions about goals, objectives and activities. Reliable monitoring requires realistic and clear project design, and may, in the case of mid-term assessment, demand adjustments in the project structure in order to secure goal achievement and fulfilment of objectives. If the project design is unclear, monitoring should identify this and assess its potential impact on objective and goal achievement.
18
Research Notes 02/2004
Second, monitoring also facilitates accountability of and transparency by the management. Transparency requires information flows. It helps the management to communicate the performance of the organisation, and to build up a project history. Monitoring may be helpful for subsequent evaluations of the project and with communication with partners. Indicators should be actively applied in monitoring and supervision of the project. Midterm reviews often neglect the importance for using indicators once identified, and how these indicators may be helpful in identifying and describing human rights changed related to the project (positive or negative) half-way through the project cycle because the “reviewer” (usually the management of the project, organisation etc) focus exclusively on output indicators. Hence, focus is on activities with little attention being paid to the fulfilment of human rights objectives. In building up a narrative of project implementation, the reviewer (usually the management) is also identifying a structure of attribution. Changes in output X are explained by input Z. Changes in objective Y are explained by or related to activities A and B. In explaining or reflecting about the causation of change, the reviewers have to interpret the processes and contexts of change caused by the project. General Guidelines on Monitoring • A project that ignores monitoring may have unclear focus or too general ambitions. The aim of monitoring is to document a process of change, and to assess whether this process can be attributed to the implementation of a particular project. Monitoring indicators are tools of management, and may enhance transparency and communication. • Progress reports should describe activities and refer to production of output. They also provide indications about processes for goal achievement and fulfilment of objectives, their limitations, constraints and achievements. Progress reports highlight changes in the context of the project which facilitate or constrain its implementation. • Monitoring reports provide a substance (and an agenda) for dialogue with partners. These reports should facilitate learning, and if carried out as mid-term reviews, the may enable adjustments of the project. Monitoring is usually an internal activity, undertaken by the project staff itself. • As a rule, effective monitoring requires baseline data (data collected prior to project) that offer benchmarks for assessing the impact of the project. One hurdle about base-line data is that they may be made less relevant by changes in the social and political environments during the project cycle. This, however, does not reduce the need for collecting baseline data in the design phase of the project. Two Dimensions of Impact
19
Observations on Human Rights Impact Assessment
An evaluation is a systematic and balanced assessment of on-going or completed activities, their design, implementation and results. The aim is to determine the fulfilment of objectives, resource efficiency, developmental effectiveness, and the impact that the fulfilment of objectives has on the overall human rights goal of the project, and its sustainability. This evaluation definition addresses the components of an evaluation. These are the design of the project, the relevance of objectives, their fulfilment, the efficiency in resource utilisation, the effectiveness in fulfilling objectives, and the effectiveness in goal achievement, that is, the impact that the project has had on human rights conditions. Effects may be intended and positive, but as noted, projects are often accompanied by unintended negative effects. Sustainability is an additional component that is important to consider in any evaluation, just as the effects on institutional capacity and competence. Human rights projects have impact at two interrelated levels. On the one hand, a project may facilitate or strengthen the capability of an individual rights-holder to claim his or her rights. This may imply that a rights-holder gets a better capacity to make a plea for litigation or other legal protection of rights. This may occur, for instance, through an awareness-raising campaign in a village. We refer to this as a human rights impact of first order. First order impact refers to the individual level. On the other hand, in making pleas, individuals very often rely on support by others, particularly non-governmental human rights organisations or public institution for the protection of the rights of the citizen. Organisational or institutional support is very often a requirement for enabling individuals to demand protection of their rights. Human rights projects that enhance the capacity of human rights organisations and institutions to respond to individual petitions have a human rights impact of second order. Second order impact is impact at the institutional level. Human rights impact assessment aims to unpack and assess the effects of a project in a social, cultural or political locality, or in a broader national or international context. Human rights impact assessment, like environmental impact assessment, relates to project assessment ex ante as well as ex post. Human rights impact assessment is therefore an activity of planning as well as of evaluation. It draws on information and data from the project itself, as well as knowledge about the political, social and cultural environment. Although it can be argued that the credibility of impact assessment in the evaluation stage requires assessments by external (independent) evaluation teams in most cases (due to resource constraints) impact analysis has to be made by the organisation or institution itself. In the latter case, an added function of impact assessment is to provide a framework for institutional control and correction. Second order impact is important for sustaining human rights impact of first order. For instance, if an organisation was unable to respond to demands by individuals who had become aware of their right through an awareness raising campaign, but did not have the resources to make effective human rights complaints, the impact of the awareness raising would not be sustainable. In other words, second order sustainability of human rights
20
Research Notes 02/2004
awareness-raising is basically institutional, and non-governmental organisations are often effective mechanisms for its realisation. Evaluation of impact can be important in terms of inspiring complementary and follow-up projects. The success of a project may trigger an expansion; failure may trigger another attempt, or the termination of support. In summary, the thrust of human rights impact assessment is to examine how a project or programme influences the general processes of human rights change. Indicators of Result in Societies in Transition In societies experiencing political transition from authoritarianism, the main objective of human rights organisations are usually to contribute to the overall goal of change. Impact assessment in these situations, requires indicators that reflect trends of change. From a human rights point of view, at least four sets of indicators are of interest: indicators reflecting the position of international human rights law (in politics and jurisprudence); indicators reflecting state compliance with political rights and freedoms; indicators reflecting state compliance with civil rights; and indicators reflecting changes in the right to an adequate standard of living (and other economic and social rights). These indicators define the ultimate goals of transitions in human rights terms, and often these goals are concurrent with the goal of human rights projects in transitional periods. They characterise the process and trends of social and political change in human rights terms (and may indicate risks of and constraints on the work of human rights institutions). The choice of key result indicators in times of political transition varies with context and the conditions of each country case. Table 2 suggests examples of key result indicators for human rights impact analyses in transitional situations.7
7
See also Hans-Otto Sano and Lone Lindholt 2000. Human Rights Indicators 2000. Country Data and Methodology. The Danish Institute of Human Rights, Copenhagen. 21
Observations on Human Rights Impact Assessment
Table 2 Key Result Indicators for Societies in Political Transition Acceptance of Compliance Compliance Trends in Human Rights Law with Political With Civil Standard of Living Rights Right Ratification of Denial of ExtraProportion of people international and participation in judicial with inadequate intake regional human the political killings and of dietary energy rights treaties process disappearanc es Reservations to Denial of Torture and Proportion of human rights freedom of ill-treatment underweight among treaties association under-five children Responses to domestic and international human rights criticism The incorporation of human rights law into domestic
Denial of freedom of expression
Detention without charge or trial
Proportion of males and females with inadequate intake of dietary energy
Denial of freedom of assembly
Unfair trial
Proportion of public expenditure on primary health care Life expectancy at birth Maternal mortality ratio Net enrolment ratio in primary education Share of public expenditure on primary education Overall adult literacy rate Gender ratios in primary education
Key result indicators include, firstly, indicators of acceptance of human rights law because these indicators establish a framework for human rights monitoring of the transition process as well as the new political dispensation. Secondly, the table includes indicators of compliance with political rights. Authoritarian governments may comply with some political freedoms, while systematically deny people their civil rights. Thirdly, the illustration includes indicators of compliance with civil rights. Typically, these indicators reflect the persistence of oppression and serious human rights violations. And lastly, indicators of trends in the standard of living are included, reflecting trends in respect to basic subsistence rights, such as the right to food, the right to education and the right to health. These indicators are important in assessing political legitimacy of the new
22
Research Notes 02/2004
regime (by assessing its popular support) as well as the capacity of people to take part in the political process.
CONCLUDING REMARKS This paper has suggested that human rights actors (state as well as non-state) should be more systematic and consistent in identifying indicators of objectives and goals in project design and implementation. This is important for documenting the impact of projects, and for analysing whether expected goals have been achieved. Indicators help to make needed adjustments of objectives along the road, and, hence, strengthen the capacity of organisations and institutions for human rights promotion and protection. Not least, indicators of objectives and goals are needed in order to evaluate the impact organisations may have on transitional political contexts, but they should always be specific and contextual, and developed for each project or policy. The overarching goal of human rights interventions is to improve the conduct of public authorities in terms of respect for and protection of rights. An equally important goal is to facilitate and improve actual enjoyment of human rights by citizens and groups. These overriding, but important goals are often neglected in project documents which tend to focus on capacity-building and awareness-raising. Indicators are rarely identified properly when it comes to the objectives and goals of project. It becomes difficult (and perhaps impossible) to assess the impact projects have on human rights enjoyment at the individual level, and on human rights compliance at institutional and national levels. There are, moreover, important limitations in impact assessment such as the problem of attribution and in measuring sustainability of projects. The scale of impact, and indicators for measuring scales, are also big challenges, although a sharper focus on the rationales for making impact assessments opens up opportunities for being more explicit about scales and levels of impact. Hopefully, the observations presented here offer a grid of issues for further debate and elaboration in order to make human rights impact assessment more systematic and reliable as to goal achievement and fulfilment of objectives.
23
Observations on Human Rights Impact Assessment
REFERENCES Jabine, Thomas B. and Richard P. Claude (eds.). Human Rights and Statistics. Getting the Record Straight. Philadelphia: University of Pennsylvania Press, 1992. NORAD. Handbook in Human Rights Assessment. State Obligations, Awareness and Empowerment. Oslo, NORAD Urban Johnsson. Human Rights Approach to Development Programming. Nairobi: Unicef, 2003. Radstaake, Marike and Daan Bronkhorst. Matching Practice with Principles. Human Rights Impact Assessment: EU Opportunities. Utrecht: Humanistic Committee for Human Rights, 2002. Hans-Otto Sano and Lone Lindholt 2000. Human Rights Indicators 2000. Country Data and Methodology. The Danish Institute of Human Rights, Copenhagen. SIDA. The Evaluability of Democracy and Human Rights Projects. A Log-frame related Assessment. Stockholm, 2000, SIDA Studies 00/03. UNDP. Human Development Report 2000. New York/Oxford: Oxford University Press, 2000. UNDP. Integrating Rights with Sustainable Human Development. A UNDP Policy Document, New York, January 1998.
24