2.6. Industrial policy evaluation

2.6.Industrial policy evaluation

Almost all of the analysis until this point was focused on the comparison or classification of industrial policies in countries. However knowing if or what some countries do or do not is still a long way from the policy-relevant conclusion on what is best to do. For this purpose some answers could be provided by policy evaluation – assessment as regards the impacts of specific policy interventions, policy packages or even overall policy frameworks.

However in the current state of socio-economic sciences such kind of evidence is scarce; even to carry out single evaluation, in a fast-changing policy and economic world is difficult and costly. Even if some evaluations are available, it is often questionable the extent to which examples from one place are applicable to another place as policy interventions are almost always carried out with modifications. There is also usually difficulty to use a control-group in policy setting, therefore compromising causality assumptions in any such evaluations. Finally, many of the evaluations are carried out only over a short period of time (during which the impact might not become visible) and must somehow take into account variety of other simultaneous interventions and asymmetric shocks. Therefore it is unsurprising that there is a lack of robust industrial policy evaluations (OECD, 2014).

There are two emerging perspectives, worth mentioning that could somehow improve the value of evaluation for policy making: First of all, it must be recognised right away that existing most rigorous evaluation methods are difficult to apply to majority of policy interventions. Evaluations using counterfactuals and control groups should be selected only for simple policy measures such as market interventions. For more complicated measures (such as policy packages or broad-based policies targeting sectors/regions) use of evaluations with counterfactuals and control groups only for those elements where they are best fit and for most complex policy frameworks these must be evaluated informally, with lots of experimentation and learning by doing.

Secondly, for evaluation to become more useful given the growing complexity of policy, it is to use it more in developmental, rather than evaluative form and design policy implementation process in a way that it allows constant feedback, frequent re-assessment and if required modification of goals as well as a multitude of continuous learning. In such format, evaluation should avoid being a one-off large project but rather a continuous flexible and lean process.

As one possible example of innovative evaluation approaches, Matthews and White (2013) proposed a developmental industrial policy evaluation method using sequential hypothesis testing. They propose to adopt a lighter alternative than traditional evaluation, using techniques originating from intelligence community. The core of the method is sequential testing of concise hypotheses (propositions) about policy status under the conditions of uncertainty and complexity and using a standard reporting template putting them together under a unified framework for analysis, with conclusions to be used for policy (re-)design.

An example of such an evaluation framework and reporting template is provided below, with five sections for testing, including the intervention logic, programme outputs and outcomes; factors contributing to outputs and outcomes; assessment of effectiveness and finally summary conclusions.

31