Measuring national statistical capacity in conflict and fragile states undp11

Discussion Paper Measuring National Statistical Capacity in Conflict and Fragile Countries This paper discusses the issue of measurement of statistical capacity in conflict ridden and fragile countries, showing its importance for public policy and measuring peace- and statebuilding strategies, and as an indicator in itself of State capacity. It critically reviews the existing tools for measuring national statistical capacity and their applicability to conflict-fragile countries, identifying shortcomings and putting forward some suggestions on how to adapt them to conflict and fragile countries

01 February 2011

Statistics and the State i

Counting is an essential function of modern states. The very notion of policy making is intimately associated to the capacity of ii iii identifying specific problems, populations, and activities . The same can be said about budgeting . Thus, the capacity to count can be included as a definitional trait of state performance.

Counting is also an essential dimension of democratic governance, in two basic senses: first, figures can be challenged by opposition parties and civil society; second, figures can –and frequently do—become a focal point for processes of public opinion sensitization. Counts produce political debates and influence the complex establishment of priorities. The upshot of this is that strengthening statistical capacities is not a purely technical process: while there are many and strong incentives to wanting to count better, “Statistics permeate modern life.

They are the basis for many counting creates grounds for contestation and debate.

The production of good data – accurate and timely —is fundamental, both for national and global decision and policy making. Since states have to dialogue permanently with national and international actors, data demands iv are increasing, and increasingly complex . However, fragile states are under particular stress in this domain, having difficulties meeting the demands for figures, and revealing critical weaknesses as well as a mismatch between needs and resources, in the process.

governmental, business and community decisions. They provide information and insight about the trends and forces that affect our lives.”

UN Secretary-General, message on World Statistics Day, 20 October 2010

Fragile States: Definition and Typology It is assumed herein that violent internal conflict is a constitutive part of the definition of fragility, because it entails the loss of the monopoly of violence by the state. Fragility, in loose terms, involves the undermining of the state‘s core functions – v vi functions that define an entity as being a State . This understanding of fragility has one simple but crucial implication : countries at low levels of development, or having a low democracy score (according to any of the existing measures), are not necessarily fragile. Thus, analytically we have four distinct cases.

1. Countries which are fragile because they have fell into a cycle of violence.

2. Countries which are fragile but have been able to avoid violent conflict.

3. Countries which are not fragile but have not been able to solve key developmental challengesvii .

4. Countries that have faced major violent events, but which are exiting fragility, and have the double task of reconstructing a statistical apparatus and of addressing the needs, demands, and characteristics of post-conflict populations (demobilized combatants, different types of victims, etc).

The tasks that each category faces with respect to the construction of a strong statistical apparatus are very different. For the first category, measuring, understanding, and overcoming (or at least taming) violence is an overriding concern. This category in reality contains two different prototypes. On the one hand, we find cases that correspond to the intuitive notion of ―failure‖, where all state functions and structures have been severely weakened or even suspended (DRC, Haiti). On the other, we have several middle income countries, which are extremely violent for different reasons (for example, Brazil, Colombia, El Salvador, Venezuela) but which are endowed with a modern technocracy and working bureaucratic structures. It is easy to understand that in the second case, motives related to violence rapidly make it to the top positions in the political agenda. Even in the case of states that face the challenge of ―purely‖ criminal violence, politicians may obtain substantial advantages from using/manipulating the figures, so as to increase/reduce the magnitude of the challenge for political purposes.

By definition, fragile-non violent countries face the challenge of having weak bureaucracies and less than full territorial presence of the state. Some countries on the lowest rung of the ladder of development have working state structures, but their lack of developmental success is likely to jeopardize their ability to staff their statistical offices. Here, the main issue is one of viii capacity. This is a problem that has been acknowledged in several documents , but not many operational solutions have

2

National Statistical Capacity in Conflict-Fragile Countries

been advanced. It is not enough to simply flag that there is a capacity deficit; it is essential to understand the hurdles that stand in the road to overcoming it. In particular, it is necessary to understand that this type of capacity is not easy to build, or to sustain (lack of infrastructure, good technocrats may leave in search of better salaries, etc.).

Post-conflict countries find a place in the last of the four categories mentioned above. Peace processes necessarily include the agreement to mute or at least tone down lingering tensions. Thus, countries that emerge from conflicts are characterized by the extreme sensitivity towards certain issues and the data related to them. On the other hand, they face very complicated and idiosyncratic data gathering problems.

Measuring statistical capacity: Strengths and weaknesses of currently proposed measures The importance of strengthening and measuring national statistical capacity has been recognized by a series of key documents that: a) identify its strategic importance; b) establish a set of principles to tackle it; c) provide operational and ix logistical instruments and solutions . Similarly, international initiatives have emerged, such as The Partnership In Statistics for Development in the 21st Century (PARIS21) established in November 1999 with the goal of improving effectiveness in reducing poverty and achieving the MDGs. In 2002, the PARIS21 Task Team developed indicators that would help track x progress of countries in building their statistical capacity . The OECD/DAC has also sponsored an effort to develop a national statistical capacity analytical framework—Managing for Development Results Capacity Scan (MfDR CAP Scan)— based on the following principles: a) demand-driven; b) low cost; c) strategic and broad-based; d) flexible and adaptable to country xi needs, context and priorities . The World Bank IDA Results Measurement System (RMS) monitors statistical capacity through xii an aggregate on a set of 15 indicators (See Table 1) .

What to measure: selection of indicators The strengthening of the statistical capacity requires the alignment of the interests of several actors – in particular, national governments xiii and major international actors . One of the basic operational expressions of the alignment of interests is the way in which priorities are resolved – e.g., responses to the what to measure question.

Some critical problems emerge as a result, for instance:

1. data gathering might not advance according to a coherent plan, but to the availability of international funds, which often depend on short term considerations;

2. results may not be useful to compare, to build time series, etc

xiv

.

The lack of clarity about measuring priorities in the exisiting analytical frameworks is often a result of the fact that these priorities are based on assumptions that either have not been spelled out, or are not necessarily compatible with other global objectives, and have not been considered from the perspective of ―priority coordination‖. For example, Table 1 provides a list of ―objective indicators‖ that allow the measurement of the statistical capacity of a country, and constitute a sort of minimum below which such capacity is considered severely debilitated. As every set of indicators acts both as a measure and as a system of incentives, the list obviously gives a premium to states that prioritize the measurement of precisely

3

Table 1 – Measuring statistical capacity according to IDA 2004 Proportion of population below $1/day poverty line Under-5 child mortality HIV prevalence rate of women aged 15-24 Proportion of births attended by skilled health personnel Ratio of girls to boys in primary and secondary education Measuring Results Statistics Primary school completion rate Proportion of population with sustainable access to an improved water source Fixed lines and mobile telephone per 1,000 inhabitants Formal cost required for business start up Time required for business start up Public financial management GDP per capita Access of rural population to an all-season road Household electrification rate

National Statistical Capacity in Conflict-Fragile Countries

those items.

But the list is not explained. Why those items and not others? It is assumed that the list is informed by an underlying, nonexplicit theory. There would be, however, other possible theories that would produce a different listing: in conflict-ridden fragile xv countries it is fundamental to capture inequalities (―vertical‖ or ―horizontal‖ ) and variables related to the regulation of property xvi rights , all items closely linked to armed conflict. Similarly, in conflict-affected countries it is also sometimes necessary to produce statistics on violence and displacement (for example, battle-related casualties, numbers of refugees and IDPs, demobilization of former militia members, etc.).

As IDA (2004) has suggested, the simplest measure of statistical capacity is the ability to produce the crucial indicators. But which are these? A reasonable assumption is that the state has to be effective at least in the measurement of the variables that are more directly related to the notion of statehood itself, that evaluate the core functions of the State. For example, the Crisis States Programme, LSE, defines the core State functions and indicators as (i) monopoly of violence (indicators: Military interventions; internal conflicts deaths; homicide rates); (ii) territorial reach (illegal economies, roads, telephone lines and xvii postal offices); and (iii) bureaucratic capacity (quality of bureaucracy, taxes) .

A state that is not able to measure these core functions can be seen as having lost the ability to orient itself in the fulfillment of them. Indeed, here there must be some variance – in the sense that each state might have a set of country-specific key variables— but the ability to evaluate the evolution of the core functions remains indispensable throughout.

At the same time, statistical capability is also a function of the resources allocated towards the very function of statistical data collection, i.e. whether there are dedicated institution(s) responsible for collecting statistical data, the levels (federal, provincial, local) at which they operate, the staff, financial and technical resources at their disposal, etc. This could well be a starting point, in fact, for assessing statistical capability through input indicators, especially in fragile environments where state institutions of all types are severely weakened, and resources are usually allocated to the fulfillment of basic humanitarian and development needs, rather than towards building up a strong statistical system. In these settings, along with the type of indicators, the number of indicators used is also key. Any set of critical measurement items should depart thus from a discussion of assumptions, consider the interests of the actors involved, and state- and peace- building priortities, as well as the capacity to collect and analyze the data. In this regard, a more parsimonious diagnostic tool with a small number of indicators, combining both direct measures of statistical capacity (staff, resources, technology etc), with indirect measures (type and quality of data actually generated) might be more suitable than the existing indicators being put forwarded for measuring the statistical capacity in other types of settings.

Qualitative versus Quantitative indicators The PARIS21 Task Team on Statistical Capacity Building Indicators proposes 16 quantitative and 18 qualitative indicators to measure a country‘s statistical capacity. It asserts that the usefulness of quantitative indicators are limited, because there is no benchmark to which compare them, the output indicators do not measure for effectiveness, and resource indicators do not provide for efficiency measures. In contrast, the PARIS21 Task Team seem to set stock on the qualitative indicators, which single out a set of basic dimensions that are evaluated by variables, in a 1-4 scale. A similar approach is taken by the CAPxviii Scan, an auto-evaluation process based on the five Managing for Development Results pillars . In this context, the evaluation of statistical capacity includes five components (statistics strategy, data disaggregation, data quality, survey capability, and xix performance measurement) and four ordered levels (awareness, exploration, transition, full implementation ). xx

The soundness of these qualitative indicators is debatable at least from the following perspectives :

a. The stability of the subjective assessments has not been evaluated (or at least there is no reference to the theme in the consulted documentation).

b. Their comparability, both time-wise and across countries, is also an issue. The questionnaires are full of doublebarreled, ambiguous, and multi-dimensional questions. For example, in response to the question ―Are buildings and xxi equipment adequate?‖ , it would not be unreasonable to expect a widely different answer in, say, DRC, Brazil, and

4

National Statistical Capacity in Conflict-Fragile Countries

Norway – notions of adequacy will be a function of the level of performance. Thus, the measure might be severely overestimating (or underestimating, as the case may be) the statistical capacity in fragile states.

c. The evaluation of the national statistical office is purported to be ordinal, but its ordinality is in question. It violates ordinality basically because it measures many different things at a time, and these things can vary simultaneously in different direction.

d. The aggregation of the different marks is made by a simple average. The absence of some kind of priority, and consequently, weightage, for different indicators, remains unexplained. Simple averages may not be a good recipe for capturing statistical capacity, as the very idea of prioritizing, which informs all the reflections of international agencies, suggests. Aggregation has to be considered carefully, as different and apparently sensible aggregation functions can yield widely divergent results.

Conclusions and Implications In the last years, the understanding of the importance of statistical capacity has advanced, and a series of principles, ideas, and forms of measurement related to its strengthening have been proposed. The fact that statistical capacity is linked both to technical and political processes has been clearly acknowledged. It is probably the case that – as in other domains— awareness is in itself an important step forward.

However, there appear to be problems and limitation in the existing tools to measure statistical capacity. The qualitative indicators currently being emphasized can be seen to violate ordinality, are ambiguous, and not necessarily comparable. The quantitative indicators are not based on theories and explicit assumptions, and there is a lack of coordination regarding priorities between different international actors.

These considerations are important for UNDP and other international agencies and donors supporting the strengthening of statistical capacity in conflict and fragile countries, or more broadly contributing to state- and peacebuilding processes. As mentioned earlier in the paper, working with the National Statistical Office (NSO) is critical in two ways:

1. The NSO as 'object' of an assessment: when assessing governance in conflict and fragile countries, it is very important to also assess national statistical capacity.

2. The NSO as 'subject' of a DG assessment: When assessing governance in conflict and fragile countries, it is very important for the NSO to play an active role. It is opportunity for the NSO to strengthen its capacity, which in turn contributes to statebuilding.

Undoubtedly, there is a clear and pressing need to tailor statistical capacity assessment methodologies to the particular characteristics of conflict-affected countries, which have to tackle a variety of capacity constraints and conflict-specific issues. xxii What data is to be collected is often a politically-charged matter ; there may be severe constraints in the process of data collection (accessibility of many areas, for instance); and the process of data collection, analysis and dissemination can be fraught with danger for those involved, especially if it threatens powerful vested interests. Furthermore, old statistical records may not be available for comparison, and old institutions (NSOs or others) may have been disbanded in the new post-conflict set-up. The ease of data collection, the resources involved, the capacity of individuals and institutions to analyse the data and present the findings, and the political will to strengthen the national statistical architecture, are all critical factors which will determine the selection and number of indicators to be used to assess national statistical capacity in a fragile environment.

5

National Statistical Capacity in Conflict-Fragile Countries

Resources UNDP Governance Assessment Portal, www.gaportal.org UNDP, 2011, Governance Assessments in Conflict and Fragile Environments: Challenges and Opportunities. Issue Brief.

UNDP and DIE, 2009, Users’ Guide on Measuring Fragility.

UNDP, 2009, Governance in Conflict Prevention and Recovery: A Guidance Note.

Acknowledgements: Authored by Francisco Gutiérrez Sanín, researcher at the Instituto de Estudios Políticos y Relaciones Internacionales, Universidad Nacional de Colombia. Inputs were provided by Javier Fabra-Mata, Shipra Narang Suri and Diana Buitrago. Jeremias Blaser, Marcin Buzanski, Bo Jensen, Joachim Nahem, Eugenia Piza-Lopez, Anna Praz, Jago Salmon and Timothy Sisk provided comments on the document.

The views expressed in this publication are the authors‘ and do not necessarily represent those of the United Nations, including UNDP, or its Member States.

Contact Information: Bo Jensen, Special Advisor, bo.jensen@undp.org or Joachim Nahem, Programme Manager, joachim.nahem@undp.org. UNDP Oslo Governance Centre.

Endnotes i

Throughout, expressions like ―data gathering and analysis‖, ―data collection‖, ―measurement‖, ―statistical activities‖ or ―counting‖ are used as quasi-synonyms. ii SRF Catalytic Fund Administration Unit. (2010). Catalytic Fund. Monitoring and Evaluation Arrangements. Statistics for Results Facility. iii Development Data Group. (2004). The Marrakech Action Plan for Statistics. Better Data for Better Results An Action Plan for Improving Development Statistics. World Bank. iv DFID. (2005). Guidance on Evaluation and Review for DFID Staff. v See Tilly, C. (1978). From Mobilization to Revolution. New York: Random House-McGraw-Hill Publishing Co.; Weber, M., Roth, G., & Wittich, C. (1978). Economy and society: an outline of interpretive sociology, Volume 1. University of California Press; North, D., Wallis, J., & Weingast, B. (2009). Violence and Social Orders: A Conceptual Framework for Interpreting Recorded Human History. New York: Cambridge University Press. vi For a detailed discussion, see Gutiérrez, F., Buitrago, D., González, A., & Lozano, C. (2010). Measuring Poor State Performance: Problems, Perspectives and Paths Ahead. LSE. vii Putzel, J. (1997). ―Policy Arena: Accounting for the ‗dark side‘ of social capital: reading Robert Putnam on democracy‖. Journal of International Development, 9: 939-49. viii Todorova, M. (2010). Managing for Development Results Capacity Scan: Implementation Review. World Bank. ix See, for example, Development Data Group. (2004); International Development Association (IDA). (2004). Measuring Results: Improving National Statistics in IDA Countries; International Monetary Fund. (2005). Statistical Capacity Building. Washington, D.C. x Laliberté, L. (2002). Statistical Capacity Building Indicators: Final Report. PARIS21 Task Team on Statistical Capacity Building Indicators. xi Todorova, 2010. xii IDA, 2004. xiii This last point is explicitly acknowledged by key papers. See, for example, Todorova, 2010. xiv IDA, 2004. xv See Stewart, F. (2001). Horizontal Inequalities: A Neglected Dimension of Development. University of Oxford, Centre for Research on Inequality, Human Security and Ethnicity. Oxford: CRISE Working Paper. xvi North, D., Wallis, J., & Weingast, B. (2009). Violence and Social Orders: A Conceptual Framework for Interpreting Recorded Human History. New York: Cambridge University Press.

For more information: www.undp.org United Nations Development Programme National Statistical Capacity inUSA Conflict-Fragile 6One United Nations Plaza • New York, NY 10017

Countries

xvii

See GutiĂŠrrez et. al., 2010.

Todorova, 2010. xix Underlying the fancy terminology there is a simple 1-4 scale. xx The majority of comments apply also to CAP-Scan. xxi LalibertĂŠ, 2002, p. 20. xxii For example, in Bosnia, fifteen years after the war ended, holding a basic census continues to be impossible as different ethnic groups conveniently brandish different numbers to make irrefutable claims about their majority or minority status to suit their political interests. See Narang Suri, S. (2009). Urban planning and post-war reconstruction under transitional administrations: The case of Mostar. Unpublished thesis. University of York, U.K. xviii

7

National Statistical Capacity in Conflict-Fragile Countries