Ria guide public administration by Maciej Kolczynski

THE REGULATORY IMPACT ASSESSMENT A GUIDE FOR PUBLIC ADMINISTRATION

Warsaw 2015

THE REGULATORY IMPACT ASSESSMENT A GUIDE FOR PUBLIC ADMINISTRATION

Warsaw 2015

Compiled by the expert panel of the IDEA for Development Foundation: Stanisław Bienias

Katarzyna Hermann-Pawłowska Sabina Kasoń

Maciej Kolczyński Katarzyna Lotko Rafał Trzciński

Alicja Weremiuk

Content editing by: Stanisław Bienias

Katarzyna Hermann-Pawłowska Graphics and layout by: Magdalena Sobczyńska Translation:

Elżbieta Myszkowska Printing by:

Place and Space Marta Nowak ul. Inżynierska 11 m. 6 05-822 Milanówek

ISBN: 978-8389437-63-1 Copy free of chargé The guide prepared within the project "Regulatory Impact Assessment Based on Evidence,

A Model for the Use of Existent Analytical and Evaluative Evidence in the Process of the Assessment of Social and Economic Effects of Designed Regulations" conducted and financed under the Programme Social Innovations of the National Centre for Research and Development Project contractors:

1. Leon Koźmiński Academy 2. Jagiellonian University 3. DCF Consulting Ltd.

4. IDEA for Development Foundation

5. HMR Strategic Counselling Partners Stanisław Mazur Jacek Radwański

Table of content Abbreviations used ...................................................................................................................................................................................................................................... 5 Abstract ............................................................................................................................................................................................................................................................................. 7

PART ONE – GENERAL OUTLINE ............................................................................................................................................................ 9 1.1. Introduction to the handbook ......................................................................................................................................................................... 9 1.2. Impact Assessment – basic terms and rules – Polish case .................................................................... 11 1.3. Impact assessment as an element of the public policy analysis .................................................. 14 1.4. Benefits of using evaluation approach in the IA process . ........................................................................ 17 1.5. Possibilities of using the handbook for IA initiatives .....................................................................................20

PART TWO – IMPACT ASSESSMENT PROCESS WITH THE USE OF EVALUATION APPROACH....................................................................................................................................... 23 2.1. Regulation as a theory of change............................................................................................................................................................ 23 2.1.1. Regulatory intervention logic......................................................................................................................................... 23 2.1.2. Recreating intervention theory in practice. ........................................................................................ 27 2.1.3. Research designs .................................................................................................................................................................................. 37 2.2. Planning and conducting research for the impact assessment process ...............................................................................................................................................................................39 2.2.1. Structuring a research problem ............................................................................................................................... 42 2.2.2. Defining a way of conducting research ......................................................................................................46 2.2.3. Conducting the research ........................................................................................................................................................48 2.3. Strategy for the use of research results in the IA process. .....................................................................49 2.3.1. Identifying key information. ............................................................................................................................................. 51 2.3.2. Defining target groups for communicating information............................................ 51 2.3.3. Selection of proper tools and a form of communicating the research results ................................................................................................................................... 52 2.3.4. Monitoring the use of research results. ....................................................................................................... 57 2.4. Examples of EX ANTE and EX POST assessment of a law conducted within the project . .................................................................................................................................................................................... 59

PART THREE – RESEARCH METHODOLOGY.................................................................................................................... 65 3.1. Methods and techniques of collecting and analyzing data.....................................................................65 3.2. Data sources-catalogue of useful databases and sources (national and foreign).................................................................................................................................................................... 67 3.2.1. Statistical data ...........................................................................................................................................................................................68 3.2.2. Governmental documents ................................................................................................................................................... 69

6 3.3. Characteristics of selected qualitative and quantitative research methods and techniques........................................................................................................ 70 3.3.1. Brainstorming.............................................................................................................................................................................................72 3.3.2. Mindmapping................................................................................................................................................................................................73 3.3.3. Logical matrix ............................................................................................................................................................................................ 74 3.3.4. Analyzing existent data (desk research). .................................................................................................. 76 3.3.5. Individual In-Depth Interview (IDI)..................................................................................................................77 3.3.6. Focus Group Interview (FGI)........................................................................................................................................... 79 3.3.7. Case study............................................................................................................................................................................................................. 81 3.3.8. Mystery client/mystery shopper.............................................................................................................................82 3.3.9. Cost-benefit analysis (CBA).................................................................................................................................................83 3.3.10. Expert panel................................................................................................................................................................................................85 3.3.11. Benchmarking . ......................................................................................................................................................................................86 3.4. Characteristics of selected quantitative research methods and techniques .......................................................................................................................................................................................................................................87 3.4.1. CATI technique............................................................................................................................................................................................90 3.4.2. CAPI technique........................................................................................................................................................................................... 91 3.4.3. PAPI technique............................................................................................................................................................................................ 93 3.4.4. CAWI technique .......................................................................................................................................................................................94 3.4.5. Auditorium survey..............................................................................................................................................................................96 3.5. Using public consultation results in the process of impact assessment .............................................................................................................................................................97 3.6. Towards causal model.................................................................................................................................................................................................99 3.6.1. Experimental approach..........................................................................................................................................................103 3.6.2. Quasi-experimental methods.................................................................................................................................... 105 3.5.3. When can we use methods based on counterfactual approach?. .......................................................................................................................................................... 112

Appendix 1. Contracting research. Description of conditions related to contracting additional research for RIA ............................................................................................................... 115 Appendix 2. Specimen documents to be used in designing and conducting research ...........................................................................................................................................................................123

Bibliography...................................................................................................................................................................................................................................................... 129 Information about authors..................................................................................................................................................................................................... 137

ABBREVIATIONS USED PIB – Public Information Bulletin WB – World Bank CATI – Computer-Assisted Telephone Interview CAPI – Computer-Assisted Personal Interview CASI – Computer-Assisted Self Interviewing CAWI – Computer-Assisted Web Interview CBA – Cost-Benefit Analysis EFTA – European Free Trade Association FGI – Focus Group Interview IDI – Individual In-Depth Interview LAIS – Legal Act Internet System LGU – local government units NEU – National Evaluation Unit CPM – Chancellery of the Prime Minister MID – Ministry of Infrastructure and Development OECD – Organisation for Economic Cooperation and Development DCS – Description of the Contract Subject RIA – Regulatory Impact Assessment (ex ante) ex post RIA – ex post Regulatory Impact Assessment/law functioning assessment IA – Impact Assessment PAPI – Paper and Pen Personal Interview RTO – Regional Territorial Observatories TR – Terms of Reference SEZ –Special Economic Zones TDI – Telephone in –Depth Interview RT – Regulatory Test EU – European Union PPL – Public Procurement Law SME – Small and medium-sized Enterprises

ABSTRACT The handbook entitled â&#x20AC;&#x2DC; The Regulatory Impact Assessment. A Guide for Public Administrationâ&#x20AC;&#x2122; has been developed within the MORE project financed under the Social Innovations Programme of the National Centre for Research and Development. The Guide is primarily aimed at making an attempt to use evaluation experiences for improving the impact assessment process with regard to state regulatory interventions. The impact assessment approach presented in this handbook assumes that this process should be based as much as possible on evidence (evidence-based impact assessment). The evidence, i.e., sound knowledge is provided by social and economic research. In Poland the results of social and economic research have been used for the improvement of quality of public interventions for more than ten years in the domain of evaluation. Evaluation has been developed in Poland for the needs of assessing the quality and effects of public interventions co-financed from EU means. The evaluation system developed for this purpose provides, in the opinion of the authors of the handbook, inspirations for the regulatory impact assessment system. These inspirations refer mostly to adapting relevant research approaches to assessing the impact of legal acts. This handbook aspires to be a practical tool which guides the Reader step by step through the process of designing and conducting research for the impact assessment of legal acts. The handbook also contains a toolkit of research methods and techniques which can be successfully used for the regulatory impact assessment research. In order to make it easier to understand the presented contents, the proposed initiatives have been illustrated by examples taken from evaluation practice. Thus the Handbook is to provide all the engaged in the impact assessment process with practical tips on how to conduct research based on the evaluation approach (i.e. evaluation experiences). The Guide is mostly targeted at officials of public administration who are in charge of preparing the impact assessment of legal acts. In addition, it can be an important source of knowledge for decision- makers, analysts, individuals responsible for the due governmental legislative process as well as for all stakeholders taking part in this process.

PART ONE GENERAL OUTLINE

1.1. Introduction to the handbook The handbook entitled ‘The Regulatory Impact Assessment. A Guide for Public Administration’ has been developed within the project ‘Regulatory Impact Assessment Based on Evidence. A Model for the Use of Existent Analytical and Evaluative Evidence in the Process of the Assessment of Social and Economic Effects of Designed Regulation’. The project has been implemented and financed under the Programme of Social Innovations through the National Centre for Research and Development. The project is aimed at the development and pilot implementation of an innovative model for regulatory impact assessment based on evidence. It is assumed that this aim will be achieved by adapting rules, methods and techniques of the public intervention evaluation to the needs of the impact assessment (IA) process in Poland. The handbook is an element of the research and development part of the project. The research concerned the development of a model for evaluation methods and public policy analysis to be used in impact assessment. Within the research part of the project, the analysis of the IA system in Poland has been conducted. The analysis concerned feasible applications of innovative solutions drawn from evaluation for the system of the law regulatory impact assessment. The international inspirations for the IA process have been also sought. The handbook ‘The Regulatory Impact Assessment. A Guide for Public Administra-

tion’ refers to the results of the conducted research, including the conclusions presented in the scientific publication entitled ‘Regulatory Impact Assessment. A Model for Poland’ (Górniak et al., in print). The model developed in the research and development part of the project will be implemented on a pilot basis in central government administration. The pilot scheme will involve progressing the process of impact assessment in compliance with the developed model. The assessment of legal acts will be undertaken both at the stage of their drafting (ex ante) and after putting them in force (ex post). Conclusions drawn from the pilot scheme will contribute to improving the quality of the handbook (in its electronic form) and will be included in the handbook as a practical illustration of its assumptions. Moreover, the handbook will provide an aid in preparing a series of trainings on how to use an innovative IA model, as well as workshops/seminars with the participation of high- level policy makers of central government administration. The workshops/seminars will aim at presentation and promotion of the model. The handbook contains a step by step description of organizing and conducting impact assessment based on an evaluation approach. The layout of the handbook includes three parts: • Part one – general outline – introduces the subject of impact assessment and presents connections between IA and public policy analysis alongside with evaluation. It also takes account of possibilities of using evaluation experience in view of IA. • Part two – refers to using intervention theory in IA, recreating intervention logic and using research designs, designing and conducting research for IA as well as a strategy for using research results in the IA process. The second part is also illustrated with practical examples of using evaluation approach in IA. • Part three – contains research methodology, including data sources as well as tools and techniques useful both at the stage of assessing regulatory impact of a draft legal act and at the stage of researching on real effects of the implemented interventions. The main part of the handbook is supplemented by the following Appendixes:

• Appendix 1. Contracting research. The description of conditions related to contracting additional research necessary for preparing IA. • Appendix 2. Specimen documents to be used while designing and conducting research for IA. The handbook is targeted at: • Officials participating in drafting a legal act, • Analysts (in and out of administration) who conduct impact assessment in view of the proposed or enforced regulations, • Decision- makers using the impact assessment results for making decisions in the legislative process, • Individuals in charge of the due governmental legislative process, • All stakeholders participating in the legislative process (e.g. participants of public consultations).

1.2. Impact Assessment – basic terms and rules – Polish case

Impact assessment, also referred to as regulatory impact assessment has been gradually introduced in Poland since 20011. The process of rooting this practice in Polish public administration has been influenced by Poland’s accession to the EU and developing in Poland the Programme ‘Better Regulations 2015’2 which assumed optimizing the regulatory impact assessment system in Poland. According to the definition of impact assessment adopted in Poland this process is described as follows:

Impact assessment is a process of analyzing possible ways of solving a problem and consequences of each of them, particularly of introducing regulations.

Source: Guidelines for Undertaking Impact Assessment and Public Consultations within the Governmental Legislative Process (the project after consultations completed on 12th December 2014)

1 2

Rules of Procedures of the Council of Ministers which were put in force in 2001

esolution No 13/2013 of the Council of Ministers of 22nd January 2013 on adopting a development R programme known as ‘Better Regulations 2015’

In other words, impact assessment supports the process of making decisions by officials and politicians as for adopting (or not) regulatory solutions and their shape. Impact assessment should provide decision- makers with at least some information on: • identified problems • options for solutions to identified problems • arguments ‘for’ and ‘against’ related to the selection of specific options In compliance with the impact assessment system adopted in Poland, the assessment includes the three elements: • regulatory test (RT) conducted at the stage of defining the assumptions of a draft legal act • regulatory impact assessment (RIA) presenting the impact assessment results at the stage of drafting a legal act • assessment of functioning a law (ex post RIA) presenting the impact assessment results at the stage of enforcing a legal act At the stage of drafting a legislative act impact assessment is mainly aimed at assessing future results of feasible alternatives to solving a problem in order to select most beneficial of them. At the stage of the assessment of functioning an applicable law, impact assessment is to assess actual results of enforcing a given regulation. The assessment conducted at this stage could lead to amending a legislative act or to drafting a new legislative act. Impact assessment is accompanied by the process of consultations which is aimed at considering opinions of all the stakeholders. The results of this process are included in a report on consultations. Nowadays the way of undertaking impact assessment is regulated in Poland by: • Resolution No 190 of the Council of Ministers of 29th October 2013 - Rules of Procedures of the Council of Ministers • Guidelines for undertaking impact assessment and public consultations within the governmental legislative process3 . 3

raft guidelines refer to another formal name: "Guidelines for Undertaking Assessment of ExpecD ted Social and Economic Results Pursuant to Art. 24(3) of the resolution No 190 of the Council of Ministers of 29th October, 2013- Rules of Procedures of the Council of Ministers". In this guide both names are interchangeable. As of 30th March, 2015 "Guidelines are under development after completing public consultations. The document itself combines the applicable Guidelines for Regulatory

The impact assessment process at various stages of drafting a legislative act results in preparing uniform specimen documents which are found in appendixes of Guidelines for Undertaking Impact Assessment and Public Consultations within Governmental Legislative Process. The specimen documents (for regulatory test, RIA and ex post RIA) were adopted by the Standing Committee of the Council of Ministers on 12th September, 2013 and shared in the Public Information Bulletin (PIB) on the website of the Chancellery of the Prime Minister (CPM). Undertaking impact assessment in Poland is obligatory in the case of drafting legislative proposals of the following normative acts: • laws (RIA and regulatory test if a draft law was prepared without a prior adoption of the assumptions), • normative acts of the Council of Ministers (RIA), • Prime Minister’s or Ministers’ decrees (RIA), • Prime Minister’s orders (RIA), • Assumptions of draft law (regulatory test). Also, the assessment of a regulation in force (ex post) must be prepared at the request of the Council of Ministers (or its subsidiary body) or if it was specified in a regulatory text, in RIA or in a legislative act itself. The Head of the Chancellery of the Prime Minister is in charge of coordination of conducting IA. At their request an applicant authority conducts RIA for a draft governmental document other than a normative act. In compliance with the provisions of Guidelines the following are in charge of the validation process of impact assessment: • Panels on Regulatory Impact Assessment (in specific ministries). • Panel on the Governmental Work Programming on whose RT and RIA acceptance depends the entry of proposals of draft law assumptions, draft laws and draft Prime Minister’s decrees to the Indicative List of Programmatic and Legislative Works of the Council of Ministers. • Department of Programming and Regulatory Impact Assessment in CPM. The process of IA preparation is based on the principles of proportionality, objectivity, transparency and publicity as well as subsidiarity. Impact Assessment of 2006, Rules of Consultations Conducted in the Process of Preparing Governmental Documents of 2009 and Seven Rules of Consultations" of 2012 and thus it replaces the mentioned documents.

1.3.Impact assessment as an element of the public policy analysis The analysis of public policies supports the decision- making process with reference to public policies by providing policy -makers with evidence in terms of sound and accurate knowledge. This knowledge is obtained in an analytical process which, depending on an adopted approach, could include a few or several steps. A popular range of actions which need to be undertaken in analyzing the public policy process is presented in Box 1. In practice, impact assessment satisfies the requireBox 1. 8 steps in the analysis of public policies: ments which result from the • Defining a problem analysis of public policies • Collecting evidence (Górniak et al., in print). Im• Designing alternatives (options) pact assessment, along with • Selecting assessment criteria evaluation and effectiveness • Assessing effects audit, is one of main analyti• Presenting trades-offs of choices cal ventures allowing to pro• Making a choice • Presenting analysis results vide knowledge for decisionSource: Bardach, 2007. making processes at the level of public administration. It refers to one sort of public interventions i.e. regulatory interventions of the state. While designing an initiative of the state in this area, the use of impact assessment for gaining sound and accurate knowledge in the making-decision process refers to the specific steps enlisted above. International experiences indicate that although the impact assessment systems in different countries vary, they have the following elements in common (Górniak et al., in print): • analyzing a problem • pointing at desirable effects • presenting various options for interventions (aimed at achieving desirable effects) and analyzing their consequences • public consultations • selecting the recommended options for an intervention • analyzing the effects of functioning an intervention

The adopted in Poland approach to undertaking impact assessment at the stage of preparing assumptions of a legislative proposal (regulatory test), at the stage of drafting a legislative act (RIA) as well as after its implementation includes the following elements: • identifying and analyzing a problem • defining objectives • defining feasible ways (options) of achieving objectives • comparing options for activity and recommending the best one of all • defining the implementation and evaluation plan • reviewing how the implemented solution worked (evaluation) These steps undertaken at various stages of preparing the regulation fit in with the assumptions of the public intervention analysis. The brief description of specific steps which need to be undertaken in case of Polish IA presented below are based on Polish Guidelines for Undertaking Impact Assessment and Public Consultations within Governmental Legislative Process. STEP 1. Identifying and analyzing a problem refers to defining what does not work, why it does not, whom it concerns and how we know about it. Problems could result from legal regulations which are already in force. In this case a source of information on the problem could be ex post RIA . The basis of commencing works on a regulation could be existent strategic documents identifying the problems and pointing at directions of initiatives which are necessary to be undertaken in order to overcome the problems and achieve objectives specified in these documents. The state intervention could be also a response to market failures. As well, in this case a thorough analysis of the problem is necessary. STEP 2. Defining objectives is a response to an identified problem. The objective indicates a desirable effect of the enforced amendment. The objective should be specific, measurable, acceptable, realistic and it should determine the deadline for it to be achieved. STEP 3. Defining feasible ways (options) of achieving objections includes the analysis of various feasible solutions to the identified problem. A very important element is the analysis of the zero option i.e. the option referring to undertaking no initiative at all. The analysis should also include non-legislative options for solving the problem.

The analysis should lead to defining costs and benefits of specific solutions i.e. it should identify certain trade-offs concerning specific alternatives to initiatives and risks related to them. The analysis should also point at the level of acceptance of the proposed solutions as a result of public consultation, at the feasibility of putting regulation in force as well as at potential unexpected consequences of the regulation. STEP 4. Analyzing costs and benefits of options for activity includes the assessment of losses and benefits related to the planned solutions to the problem. The analysis refers mainly to regulatory impact assessment in terms of public finance sector, labour market, economic competitiveness and entrepreneurship as well as society. The analysis should include both monetary and non-monetary costs and benefits of the considered solutions and in this respect it should be based on reliable data. STEP 5. Comparing options for activity and recommending the best one - it is based on a prior cost-benefit analysis of options for activity. If a regulatory option is selected it is requested to justify why the identified problem cannot be solved in a different, non-legislative way. STEP 6. Defining the implementation and evaluation plan refers to planning subsequent steps of enforcing amendments to a regulation in the form of a schedule. The implementation and evaluation plan should define how achieving the regulation objective is to be measured as well as predict a system of monitoring and evaluating the assumed effects. STEP 7. Reviewing how the implemented solution works (evaluation) includes the assessment of the extent to which the assumed objective has been achieved, the assessment of the measures undertaken for solving the identified problem, the evaluation of regulatory impact which was assessed as a result of RT as well as the evaluation of real regulatory assessment. The evaluation should also serve for the identification of problems related to introducing a given regulation and include recommendations with regard to solving the aroused problems. This stage could be a starting point for further works on the regulation. In this case these results should be used in a regulatory test for a new draft legal act. It is worth noting that although Guidelines for Undertaking Impact Assessment and Public Consultation within Governmental Legislative Process treat

evaluation as the assessment of functioning a law after it is put in force (at the stage of ex post RIA ), the approach adopted in this handbook indicates possibilities of using evaluation experience at each IA stage (also RT and RIA). The elements of impact assessment presented above are put into effect at various stages of enforcing a legislative act and are included in the documents prepared by administration which are a product of the above analysis conducted for regulatory test, RIA and ex post RIA.

1.4. Benefits of using evaluation approach in the IA process

The analysis of the effects of public interventions (at the stage of both expected impact and real effects of authoritiesâ&#x20AC;&#x2122; initiatives) is a necessary element of conducting public policies. Developing systems of the assessment of regulatory intervention impact as well as the evaluation systems of public interventions serves the purpose of the analysis of the effects of public interventions. There are many similarities between both systems, which are presented in Box 2.

Box 2. Similarities between evaluation and IA Both impact assessment and evaluation: 1. refer to the assessment of the state interventions (financial or regulatory) 2. result in creating a new knowledge 3. aim to define intervention effects by using scientific methods 4. conduct the state intervention assessment at the stage of both planning it ex ante) and after implementing it (ex post) 5. increase effectiveness, efficiency and utility of public interventions 6. are processes, the progress of which is within the scope of responsibility of officials who should possess relevant expertise and competence 7. are positioned in the same institutions (most frequently ministries) 8. face a challenge of the relevant use of their results 9. are under threat of lack of objectivity due to assessing the results of their own work Source: Bienias et al., in print.

In view of the similarities presented above there is a possibility of making use of mutual experiences in the progress of these processes by public administration. In case of Poland the evaluation system, initially created within the EU Cohesion Policy in the area of using the EU funds, has been developed to such extent that it is possible to adopt effectively the experiences related to the evaluation process to the needs of the impact assessment system. The key evaluation inspirations for impact assessment in Poland which are identified and presented in the publication ‘Impact Assessment. A Model for Poland’4 are shown in Table 2. Although the context is Polish, these examples may be also useful inspirations for adopting experiences stemming from evaluation to impact assessment systems in other countries.

Table 1. Evaluation system inspirations for the impact assessment system in Poland Institutional system

• Clear positioning of competence with regard to coordination of the evaluation system in the National Evaluation Unit (Ministry of Infrastructure and Development- MID) which sets the rules for conducting evaluation and coordinates the whole process. Stakeholders of the process • A wide range of the process stakeholders (the European Commission, representatives of monitoring committees and representing society, experts, academics, the Polish Evaluation Association). Administrative capacity • Introducing systemic solutions strengthening administrative capacity i.e. an annual evaluation conference, the Evaluation Academy, the Panel on Evaluation (a forum for exchanging experiences and knowledge with regard to various practices). Capacity of external institutions • C reating a strong market of external entities conducting research commissioned by administration with regard to the needs of evaluation Methodology of research • Using differential methodology relevant to the needs of research on the impact of (financial) interventions

In a chapter concerning the application of inspirations from evaluation for impact assessment.

Use of knowledge • I ntroducing systemic solutions with regard to the information flow, including a data base built for collecting all evaluation reports • U sing a model of participatory evaluation improving the use of evaluation research results Based on: Bienias at al., in print.

The presented above practices and experiences developed with reference to evaluation could be an inspiration for the system of impact assessment functioning in Poland in the following aspects5: • Clear defining a role and competence of particular institutions engaged in conducting impact assessment. Like in the case of the institutional evaluation system, it would simplify the system and could influence improving the quality of its functioning. • Positioning the competence in terms of undertaking IA in ministries and central offices in specific units (modelled on evaluation units or based on evaluation units) would allow to build lasting institutional structures. • Separating units responsible for IA analyses in an office from the units responsible for drafting a regulation would lead to increasing the independence of the whole process and in consequence it would raise its objectivity. • Opening the impact assessment process to various stakeholders, including those from the outside of public administration (e.g. experts, academics) and outsourcing IA research would allow to improve the quality of this process as well as creating foundations for conducting debates based on evidence (research results) on the effects of planned and conducted regulations. • Using evaluation experiences in terms of building administrative capacity would allow to create more rapidly the analytical capital in Polish administration for the needs of internal analyses within institutions as well as the needs of the analyses fully or partially outsourced. • Impact assessment can take advantage of a large market of external entities conducting research commissioned by administration. Outsour5

The presented proposals are included in Bienias et al., in print.

cing the research allows to ease the burden of analytical tasks on employees dealing with substantive aspects and makes it possible to gain necessary expertise. • Impact assessment can take advantage of a variety of research approaches used in evaluation, particularly theory-based approaches. Their application to the impact assessment process requires the use of a wide range of research methods and techniques both quantitative and qualitative . • Impact assessment can take advantage of the experiences of participatory evaluation allowing for close cooperation between entities responsible for preparing a regulation (e.g. between a legislative panel) and officials responsible for conducting IA (a research/analytical panel). The participatory model allows for an effective and efficient use of the results of the conducted research. • Finally, the use of evaluation experiences with reference to creating systemic solutions for enhancing the application of the research results (the system of implementing conclusions and recommendations, evaluation database) concerning IA as well as the integration of various databases would provide conditions for conducting a based-on-evidence debate on the effects of public interventions. The above proposals indicate that the use of experiences in building and managing the evaluation process in Poland have many possible applications to the IA system being in development. They may be also successfully adapted to different national contexts regarding functioning of IA and its relations to evaluation systems in other countries.

1.5. Possibilities of using the handbook for IA initiatives

The handbook is a tool for fostering the impact assessment process by application of evaluation experiences in the impact assessment process. The handbook takes advantage of the above presented inspirations and fits them into the practice of conducting impact assessment in the following scope: • It promotes organizational separation of the process of drafting a legal act from its assessment.

• It simultaneously indicates a possibility of using participatory evaluation experiences for close connecting the process of drafting a legal act to the assessment of its impact (at the stages of RT and RIA). • It recommends basing IA on sound evidence and results of the conducted research. • It indicates possibilities of using the evaluation approach to designing and conducting research for IA. • It indicates possibilities of using the capacity of external institutions for conducting research for IA (outsourced research beyond administration). • It takes advantage of the experiences of the evaluation system created for the EU Cohesion Policy with reference to preparing a strategy for the dissemination of research results as well as monitoring the application of the results of the conducted research. • It takes advantage of a number of research approaches used in evaluation which are feasible to be used in IA, particularly theory-based approaches. • It presents research methods used in evaluation, particularly experimental and quasi-experimental methods allowing for the assessment of real effects of the implemented interventions.

As a result of the adopted approach to IA, this handbook: • • • • • •

Allows for relevant planning and progressing the impact assessment process Indicates possible ways of recreating intervention logic as a starting point to conducting IA Presents a way of verifying intervention logic with the use of research plans which facilitate designing and conducting research for IA Presents step by step the process of designing and implementing research for IA contracted or conducted internally Provides practical tips referring to outsourced research Indicates possible strategies for the diffusion of research results in the IA process

• • •

Indicates possible ways of monitoring the application of research results to the IA process as well as to conducting an evidence-based debate Includes a methodology toolbox with regard to research tools and methods to be possibly used in the IA process Provides practical examples (practices within the project, RIA and ex post RIA research) as an illustration of the process of implementing research with the use of a theory-based approach.

PART TWO IMPACT ASSESSMENT PROCESS WITH THE USE OF EVALUATION APPROACH 2.1. Regulation as a theory of change 2.1.1. Regulatory intervention logic According to the guidelines for conducting the assessment of expected regulatory impact 6 ‘Impact assessment is a process of analyzing possible ways of solving a problem and consequences of each of them, particularly of introducing regulations‘7. In view of this, one of the key tasks facing impact assessment is to define whether introducing a given normative act will allow to solve identified problems. From a bureaucratic perspective, drafted regulations could be perceived as a set of initiatives and procedures related to each other. Their selection and form are a result of the creative work of the authors of a given regulation. The objective of the introduced solutions is to make a defined change observed at the level of social and economic reality. Regardless of the problem area, created solutions are always a kind of theoretical structure referring to how a given intervention will affect obtaining this desirable change. A typical situation in the process of drafting legal acts or, in a broader sense, in designing public policies, is to assume that a given public initiative (occurrence A) is a necessary 6

uidelines for Conducting the Assessment of Expected Social and Economic Results under Art.24(3) G of the resolution No 190 of the Council of Ministers of 29th October, 2013 – the Rules of Procedures of the Council of Ministers, Ministry of Economy in cooperation with the Chancellery of the Prime Minister (as in March 2015) Ibidem, p.30

condition (often also sufficient) for making some changes at the micro level (occurrence B) which are most frequently identified directly at the level of entities which are subject to a given regulation. Subsequently, the obtained changes - in a broader perspective and after achieving the critical mass – are to also bring notable effects at the macro level (occurrence C), which are usually identified with solving the problems giving rise to drafting a given legal act.

Example 1. An example of logic: if A then B, if B then C; is public support given to economic entities under the Law of 20th October, 1994 on special economic zones (Journal of Laws of 2007,No 42, item 274 and of 2008 No 118, item 746). The law was a response to negative effects of the transformation of the social and economic system in Poland such as an increase in unemployment as well as rising disparities in the level of economic development of particular regions of Poland. The solutions provided in the law are to help eliminate these problems by creating favourable conditions for running business in areas specially authorized for this purpose. One of the basic incentives for entrepreneurs in this respect is tax exemption granted to the entities authorized to run business in the area of special economic zones (SEZ). The law and the supplementary implementing acts constitute a complete intervention (occurrence A) which is to translate directly into functioning and developing particular enterprises running business in the area of a given zone, especially it is to encourage entities to make investments and employ new staff (occurrence B). At the same time it is expected that the public aid granted to ‘zone’ enterprises will be translated not only into their functioning but it will also bring real benefits for regions where investments are located and finally for the whole country as well (occurrence C).

Obviously, the logic presented in the above example is a simplification of the rules according to which the indicated regulation works. In reality, relationships between occurrences A, B and C are not unambiguously (i.e. the cause and effect relationship) defined, as they interdepend on many factors related to one another. Public interventions, as a rule, are complex initiatives undertaken in a complicated social, economic, legal and institutional environment. To put the scheme ‘A causes B and B causes C’ actually into operation and thus to make

justified conclusions on the mutual interacting of the occurrences, additional assumptions and conditions must be fulfilled. In fact, conducting each intervention means adopting assumptions that a given legal solution will be effective (i.e. it will actually bring a desirable effect):

Example 2. In response to the labour market problems (high unemployment), the amendment of the Law on promoting employment and of labour market institutions and some other laws of 14th March 2014 (occurrence A) assumes to increase the impact of the labour market policy on the labour market situation (occurrence C) by, among others, launching mechanisms increasing the work efficiency of offices (occurrence B). The changes introduced under the law refer to the correction of the way in which the system of supporting the unemployed has worked so far (the system of conducting an intervention) with the use of existent instruments such as the Fund of Labour (foreseen resources). The changes introduced gradually (within a defined time) will be assessed two years after the entry of the amendment into force (evaluation/ex post RIA).

• in the adopted system of conducting an intervention • with reference to expected resources/regulatory intervention scale • at a defined time • in a given economic, social, institutional and legal environment. One of the basic assumptions in making a decision about the necessity of putting a given regulation into effect is stating that the intended result (the problem solved) cannot be achieved through non-regulatory initiatives. Then detailed assumptions are made regarding the accuracy of means selected to bring the expected effects of the regulation. These assumptions define in the most general way that the form and scope of initiatives planned to be undertaken are capable of solving the identified problems. Simultaneously, it is assumed that there is a possibility of rooting a given regulation in a broader applicable institutional system (legal, social, economic) which is the background for enforcing legal amendments and triggering desirable changes. In other words, it

is assumed that all factors related to the background of implementing a given regulation will not have an important negative influence on its functioning nor will they disrupt its putting into effect. The assumptions adopted at this stage refer also to possible ways of circumvention of a given regulation and thus possibilities of avoiding its potential impact. Diagram 1. Assumptions in the process of implementing a regulation â&#x20AC;&#x201C; generic model Assumptions

Problem demanding state intervention

Assumptions

Regulation

Assumptions

Observed change

Actions

Assumptions

Source: by authors

An important thing is that the most of these assumptions are not expressed in an explicit way in terms of the regulation itself. They are often on the minds of decision-makers and people in charge of public interventions (Pawson and Tilley, 1997). Usually in an official (public) circulation there is a very simplified scheme of a given regulation, referring to the already mentioned â&#x20AC;&#x2DC;A causes B and B causes Câ&#x20AC;&#x2122; (or in other words: if we introduce a given regulation, the problem will be solved). However, the assumptions typical of this scheme are usually omitted, so it is supposed a priori that they will be fulfilled. This situation refers to expected effects as well as potential unintended results (indicating that unintended results will not just be brought is not rare). However, if it is assumed that achieving the objections of a given state intervention depends actually on other factors as well as on the structure of a given regulation itself, the knowledge about the assumptions to be fulfilled for achieving a goal of an undertaken initiative is really of key character in view of the assessment of a given legal solution. Taking this into consideration, one of the tasks facing regulatory impact assessment is to define a logic of the analyzed intervention (regulation), indicate critical assumptions on which the proper work of the regulation depends and assess to what extent these assumptions (ex ante assessment) could be fulfil-

led. The assumptions analysis should in the first place refer to the relationship between elements of the planned intervention and expected results (e.g. in the case of SEZ it is assumed that tax exemptions will translate into new investments leading to creating new jobs in the zone area). Analysis at this stage should take account of identification of unintended effects (as well undesirable ones). Secondly, a key element is to verify the assumptions related to external factors, including the background, which have a potential impact on functioning (effects) of a given regulation.

2.1.2. Recreating intervention theory in practice

Theoretical frames for the analysis of intervention theory are outlined by a theory-based approach. This approach, like the other methodological concepts presented in this publication, is used and developed with reference to evaluation research (Leeuw, 2012). The development of it has been a response to the so-called â&#x20AC;&#x2DC;black boxâ&#x20AC;&#x2122; problem which appears in research focused on defining only quantitative results of an intervention. In fact, the knowledge of what are the quantified effects of a given public initiative is of a key character in view of the assessment of its effectiveness. However, the knowledge itself hardly explains why the effects of the particular nature have been brought. This knowledge is undoubtedly indispensable in view of optimizing conducted interventions. Only by identifying the causes which determine that a public initiative is or could be effective, does it allow for making corrections leading eventually to more effective management of public resources. (Martini, 2011). In view of the above, theory-based research is in the first place focused on discovering mechanisms which, according to the authors of public policies, make designed initiatives effective. Secondly, the research makes an attempt to test the defined intervention logic, i.e. it assesses to what extent the adopted theory is correct (GĂłrniak, 2007) and points at real chances of putting it into practice. While assessing it, all factors which could possibly affect the process of functioning a given theory are taken into consideration (Leeuw, 2012). As mentioned above, public interventions such as programmes, projects, regulations are rarely based on a clearly defined theory. It also refers to defining

expected effects of particular initiatives planned in view of a given regulation. At the same time there is no critical reflection on the assumptions determining the proper work of the regulation. Therefore a starting point to each research process which takes advantage of a theory-based approach is the need of recreating the theory (logic) of a given initiative. This stage of the research is mainly of a conceptual character. In the first place it includes defining a complete mechanism or mechanisms of how a given regulation works and points at all critical assumptions for its functioning. It is worth bearing in mind that this process is usually iterative and the final logical model is developed in the progress of the research and updated several times, as a result, among others, of arrangements between key parties engaged in functioning a given regulation. Ultimately, a logical model of regulation (intervention theory) should be approved by all the parties to whom a given solution concerns. After the conceptual stage of the theory-based research there is a typically empirical stage which involves testing the recreated logic. It is essential that the process of testing intervention logic should be differentiated depending on whether RIA is prior to introducing a regulation (ex ante) or whether it makes attempts of assessing its actual functioning (ex post). In the former case testing will be also to much extent conceptual and oriented on a critical analysis of the regulation assumptions without verifying how they work in practice. In view of this it is necessary to make a prediction about to what extent the adopted assumptions for a given model will remain unchanged at the time when a given regulation is in force. If possible changes are predicted in this respectit is necessary to define how the assumptions transformed or outdated will possibly affect the functioning of the analyzed regulation. With regard to ex post RIA, it is possible to actually verify the correctness of the recreated theory of the introduced regulation. In this respect it is advisable to make an attempt of defining real (i.e. in the terms of the cause-and-effect relationship) effects ofthe functioning of a given regulation (more in the subsection: Towards a causal modelâ&#x20AC;Ś..).

Diagram 2. Generic diagram of theory-based research Prospectively-oriented ex ante analysis Recreating regulation logic

Testing regulation logic Ex post analysis verifyling theory through prism of real regulation effects

Conceptual stage

Empirical stage

Source: by authors

The description presented below focuses mostly on the first stage of theorybased research, namely on issues and methods related to recreating intervention theory. The second stage is described in subsequent parts of the handbook. The ideas developed in relation to theory-based approach, mostly with reference to evaluation research, have led over the 40 recent years to developing at least a few specific methodological approaches. Among them, the following ones should be particularly distinguished: theory-driven evaluation (Chen and Rossi, 1980), theory-based evaluation (Weiss, 1995) and realistic evaluation (Pawson and Tilley, 1997). Despite some differences as for emphasizing selected elements of the approach, these concepts have one thing in common which is treating every public intervention as a theory. They are also similar in discovering and expressing these theories. In this process three key elements could be distinguished in particular: 1. creating a causal chain of intervention (intervention theory) and identifying assumptions which influence its effectiveness 2. analyzing the background for conducting an intervention 3. areas/dimensions in which effects of public initiatives could be differentiated A causal chain of intervention (intervention theory) and identifying assumptions which influence its effectiveness The mentioned causal chain links planned initiatives to intervention results and expected effects (White, 2009). At this stage it is necessary to indicate what problem is to be solved, how, step by step, the implementation of a intervention

has been planned (how problem is to be solved), including its indirect steps and their direct results. Finally, it is advisable to indicate how these results translate into expected regulation effects and thus solving the problem. It is essential to indicate all potential factors or causes of the expected changes. These changes could particularly refer to the way of working (functioning units) the entities which are subject to a given regulation. Caroll Weiss indicates that it is essential to distinguish all key mini-steps which build up a characteristic causal chain in which some occurrences are consequences of others as a result of introducing a given solution (Weiss, 1995). In practice, the main source of data necessary for defining such a chain should be a regulation itself (a draft legal act) understood as the first step. However, it is quite likely that the causal mechanism based on the regulation will be incomplete, especially if the analyzed regulation is a law and its clarifying involves issuing particular decrees. Similarly, if the analyzed regulation is related to other legal acts it is necessary to conduct a complex analysis of all law sources related to one another. The analysis should also include all kinds of strategic documents indicating a need to conduct one or another regulation as well as, if accessible, prior expert reports, research and publications on concepts with regard to introducing defined changes in the legal system. With reference to these documents it will be possible to discover particular intentions of making a decision about introducing regulatory changes in a given area. After analyzing accessible existent data (desk research) it will be also necessary to contact the authors of a given regulation and individuals who had a significant influence on it. The purpose of such a in-depth analysis is the above mentioned attempt of discovering all intentions /the cause-and-effect relationships between elements which are not expressed explicitly in the regulation. At this stage it is advisable to define which initiatives of the analyzed logic are of key character (are necessary to bring the effect) and which ones are only auxiliary. The effects of this research stage are most frequently presented in a form of graph in which all causal progressions leading to specific initiatives, results and effects are marked. Solutions referring to mindmapping techniques could be useful for creating such graphs. Below there is a generic graph for the above

mentioned example of the law on special economic zones. This time the causal chain has been described in a bit more detail. Taking into consideration accessible provisions of the law and official information on presenting SEZ 8 it is possible to state that the support given to enterprises within the zones (particularly tax exemptions) is to result in attracting new investments (i.e. investment capital flow) to authorized areas of the country. These investments are to bring a direct result which is an increase in employment in the mentioned entities. At the same time it is expected that the investments will generate the companiesâ&#x20AC;&#x2122; demand for products and services of subcontractors, including local ones. An equally important expected change is the development of new products or services in the supported enterprises, especially the implementation of technological innovations and R&B-based solutions. The changes in the enterprises are to be a springboard to more significant changes notable in a close environment of the entities located in SEZ. The changes include especially the development of local subcontractors (thanks to cooperation with the supported entities and in relation to an increased demand for products and services from the employed in the supported companies) and the development of local infrastructures (e.g. in information technology and transport). These changes made due to the economies of scale seen in a long â&#x20AC;&#x201C;term perspective should lead to an increase in employment not only in the supported enterprises, but also in other entities of the region. As for the increased demand and the development of infrastructure, they are to influence a rise in general investment attractiveness of a given location, which should encourage subsequent investors (not only in the zones). All the changes altogether should result in the social and economic development of a given region (and in view of the law scale the whole country as well) which should involve decreasing the rate of unemployment, expanding selected industries and increasing exports. In other words, putting a logical model of intervention into practice should lead to solving problems which were a cause of conducting the presented regulation. E.g. Portal of the Ministry of Economy with an exclusive section on SEZ: http://www.mg.gov.pl/Support+ entrepreneurship/Support+financial+ investments/Special+zones+ economic/ Effects+SEZ, available on 1st April, 2015 8

Diagram 3. Causal model of support within Special Economic Zones Incentives in form of public aid (income and property tax exemption)

Access to improved land, technical infrastructure and support of investments

1 Entrepreneurs make investments in SEZ under negotiated conditions of authorized operation in a given zone

Direct employment increase

Increased demand for local products and services Launching new products and services as a result of investments (including R&D)

Long-term increase in employment

Development of local subcontractors 8 Development of local infrastructure (transport, IT sector, etc.) 7

9 10

New investments in the region

Competitiveness increase of supported entities

Social and economic development of the region and the country: fall in unemployment, development of some industries, exports increase

Source: by authors

The presented model describes a situation which assumes that particular initiatives and expected results follow each other and are in direct causal relationships. In reality, as mentioned above, the adopted theories can be only put into practice provided that defined assumptions are fulfilled. These assumptions are not included in the regulation or in the documents directly related to it. Therefore identifying and presenting the assumptions whose fulfillment is necessary in view of putting the adopted theory of change into practice is part of the process of recreating an intervention theory. In the above SEZ example these assumptions are marked in the diagram as numbered rectangles. The first assumption (number 1 in the diagram) refers to the significance and attractiveness of facilities generated by a regulation in view of decision-making processes which progress in enterprises. The correct functioning of the law on SEZ forces to adopt the assumption that the conditions of income tax exemption will be favourable enough at least to some enterprises (potential investors) to actually influence their decision about selecting the investment location. Practice may turn out to be different from it. There are some situations that Poland and selected SEZ compete with other countries and locations for attracting new investments. In view of global economy, tax exemptions in Poland may turn out an insufficient incentive for desirable investors. Also, the second assumption adopted at this stage referring to relevant selection of the SEZ areas (number 2 in the diagram) may be false. Again, practice may prove that the offered investment land will not be attractive for potential investors. Bringing investors to some locations lacking in infrastructure and technical resources which are at the far side of big cities could be a real challenge despite the conveniences provided by the law. However, if the law is to erase

differences in the development of particular regions it is necessary to assume that investment will be also made in the most disadvantaged locations. The question arises – to what extent is this assumption true (possible to be fulfilled) within the adopted regulatory frame? Verifying the correctness of this assumption as well as the other ones should be conducted at the stage of testing the adopted theory of change. The third assumption is related to creating new jobs in the supported entities. Creating new jobs is required from all entrepreneurs who are authorized to run business in the zones, however, in practice it is necessary to ensure that this requirement has been met. In fact, there is a chance of improving the situation in local labour markets only if the supported enterprises hire local employees. However, it is not always true, especially if the area where investments are to be made is deficient in job seekers who are knowledgeable professionals (with defined expertise). In the end, potential employees must be interested in work for the enterprises which made investments in SEZ. In this respect, conditions offered by the supported entities will be of great importance. As it is seen, the success of the adopted mechanism of generating new jobs thanks to the supporting instrument which is SEZ is based on many assumptions. In view of intervention effectiveness (both ex ante and ex post) their verification should be the heart of conducting impact assessment. To continue, it is worth describing briefly other assumptions related to the logic of functioning SEZ. They are as follows: Assumption 4: Entities running business in SEZ use local products and services and are aimed at cooperation with local SMEs. On the other hand, local entities have capacity and competence for cooperating with the entities operating in the zones. Assumption 5: Conducted investments are in fact related to innovations – they stimulate R&D projects and transfer of new technologies. Assumption 6: The entities’ strategy assumes that they will stay in the region for a long time. Investments bring lasting effects (exceeding the assumed periods of durability). Assumption 7: The supported entities’ strategy refers to the growth theory by implementing innovative solutions. At the same time SEZ beneficiaries possess a sufficient capacity and resources for operating in a long term.

Assumption 8: The scale of conducted investments and accompanying changes is sufficient for an impact on real investment attractiveness of the region. Assumption 9: The scale of changes in the labour market as a result of the investments conducted in the zone, including a number of created jobs in particular, is sufficiently large as compared to the whole labour market (SEZ impact will be noticeable). Assumption 10: Other factors â&#x20AC;&#x201C; political, economic, legal, etc.- will foster desirable changes or at least will not be barriers for them. It should be pointed out that the above theory of change and the identification of key assumptions was mainly based on the analysis of existing data (desk research) in which documents directly related to the regulation (a law, decrees) were used in the first place. In view of creating a possibly complete frame of the theory of change the analysis of existing documents should also take into account existing research, evaluation as well as academic literature. Additionally, after conducting desk research the qualitative stage of research should follow in which recreated theory is confronted with the main stakeholders engaged in the process of SEZ functioning (representatives of government administration, companies managing SEZ, enterprises, LGUs and social partners). The process of the correctness verification of the adopted theory of change should be subject to social consultations which accompany a legislative process. The process of recreating intervention logic can be also seen through the so-called logical matrix. Its lines include subsequent levels of expected changes resulted from implementing a given project. The table columns depict subsequently: the areas of expected changes, sources of information on whether the expected change actually has been made, assumptions which determine defined changes or effects. Making changes at each level allows for making changes at a higher level (Kierzkowski, 2002). According to the generic model of designing and implementing projects, appropriate resources (financial or human input, etc.) are transformed into specific initiatives (e.g. providing access to investment land under favourable conditions) and results (e.g. retraining the unemployed) which, in turn, could be transformed into appropriate effects (e.g. a decrease in the unemployment rate in the supported group) being an actual objective of a given intervention.

Table 2. Scheme of the intervention matrix Objective

Achievement indicators

Information sources

Assumptions

Effects Results Products Input/Initiatives Source: based on Kierzkowski 2002.

Analyzing the background for conducting an intervention Understanding the background for implementing a regulation is of key significance in view of understanding a potential impact of a given public initiative. It is also important with reference to methodology of planning impact assessment. The background means social, political and economic conditions for implementing a given regulation. It is particularly essential to identify these background elements which could have a potential impact on the effectiveness of a given regulation. In fact, a regulation can function in a different way within different backgrounds. Referring to the SEZ example, it may turn out that its implementation will bring desirable results, but not always in the areas where they could be expected. For example, it may turn out that its effectiveness is strongly influenced by the level of social development of a given region. Understanding the background needs taking a deep insight into the specificity of implementing a given regulation as well as combining its elements with the indicated factors. Identifying areas/dimensions in which effects of public initiatives could be differentiated Understanding the background is a starting point to identifying the differentiated impact of regulation, which is particularly significant in view of the materialization of the recreated theory of change. There are likely situations in which a single regulation does not influence homogenously on all concerned units. In extreme cases it may turn out that the adopted intervention logic works really in a correct way in some of the units, whereas in the others the adopted assumptions fail. Making an effort of identifying a group of regula-

tion ‘addressees’ for whom the impact may be differentiated should be the next step in impact assessment using a theory-based approach. Other regulatory actions influencing given regulation (e.g. changes in national rules of state aid in the EU) should be taken into account. Apart from the above, the analysis of the regulation impact differentiation should take into consideration a territorial aspect of the implemented interventions as well as a time horizon. The moment of measuring the regulation effects is of great significance. It may turn out that a regulation ill-functioning at the beginning could bring the expected effects only after some time since its introduction (and vice versa – a regulation may work worse after some time because of finding by its recipients existing gaps and imperfections). Identifying a potential differentiation of regulation impact will be especially important in view of conducting the costs/benefits analysis. It will also influence the process of IA results’ generalization on the whole population. Recreating logic of complex regulatory intervention The example of the analyzed regulation concerning special economic zones shows a cohesive concept of public intervention. A more difficult challenge may be recreating the intervention theory in the case of complex and multi-dimensional legal acts. It refers to conducting the impact assessment of the Law of 14th March 2014 on amendment of the law on promotion employment and labour market institutions as well as some other laws. In this case we are not able to identify a cohesive theory of change but rather a few various interventions for one broader social and economic objective (a rise in employment). This situation is frequently encountered in conducting evaluation research within Cohesion Policy. For example, while evaluating a regional operational plan we need to assess a number of various interventions leading to one macroeconomic objective. In such a situation the intervention should be ‘divided’ into a number of smaller but cohesive components. Logic of particular theories of change should be recreated in a simple way separately as well. Then the theories which seem to be of key importance should be selected (those which potentially generate the most positive or negative social and economic effects). The selected components should undergo a complete assessment process and then the synthesis of results is conducted. In practice, a great deal of conducted research is based on a multiannual plans of evaluation research within Cohesion Policy.

39 2.1.3. Research designs9 A research design is a comprehensive research logic, searching and providing reliable and credible answers to research questions. Source: Olejniczak and Wojtowicz, in print.

Research designs for searching and providing answers to research questions are useful for testing intervention logic. A selection of research design depends on the questions asked in the IA process and they, in turn, depend on informative needs of research recipients. The main research designs are presented below:

Meta-analysis (systematic review) The design logic involves gaining knowledge based on a synthesis of prior research concerning a given area or topics. Meta-analysis may help understand the background â&#x20AC;&#x201C; what the situation is like in a given area, what we know about the mechanisms used so far and intervention effects. It also allows for gaining knowledge of the experiences of other countries (what worked out/why) This research design is useful at the stage of RT and ex ante RIA as well as ex post RIA. Experiment, quasi-experiment The logic of this design involves defining intervention effects by selecting two comparative groups within one population (one is included in an intervention, the other one is not) and comparing the situation before the intervention to the situation after the intervention in the two groups. The experiment allows for defining a real intervention impact. This research design can be mostly used at the ex post RIA stage or within pilot schemes conducted before drafting a legal act with the aim of analyzing alternative solutions and costs generated by them. More information on experimental and quasi-experimental methods can be found in the third part of this handbook. Statistical study The logic of this approach involves impact assessment by using statistical methods (econometric models) for defining relationships and dependencies be9

This subsection is based on: Olejniczak and Wojtowicz, in print; as well as: Olejniczak et al., 2014.

tween an intervention and expected changes. It allows for forecasting market, economic and social phenomena which emerge as a result conducting a given intervention. This approach allows to define intervention impact at the stage of both ex ante RIA and ex post RIA.

Simulation game The design logic involves recreating an intervention in laboratory conditions in order to get to know mechanisms of change. A simulation game allows for simulating real mechanisms in safe conditions. As a result, we gain knowledge of how and why the change is made following an intervention. The simulation game can be used at the ex post RIA for investigating whether and why a mechanism has worked or for anticipating the impact of a given intervention or analyzed alternative solutions at the RT stage.

Theory-based evaluation The logic of this approach involves recreating an intervention theory or the cause-and-effect assumptions as well as its verification in reality. Theory-based evaluation allows for discovering mechanisms of change but also for investigating the impact of planned and implemented interventions. It can be used at the ex ante RIA and ex post RIA stages. More information on this approach and its practical application to regulatory impact assessment has been presented in the previous subsection of the handbook. Case study The logic of this design involves an in-depth analysis of one case (a social group, a project, an institution) in reality, in the natural background and conditions. The case study allows mainly for gaining knowledge of the mechanism of change. It can be used in the first place at the ex post RIA stage.

Participatory approach This research design involves gathering opinions of various parties to whom an intervention concerns, weighing them up and working out a consensus regarding the problem as well as drawing conclusions. It allows for getting to know processes of intervention and its assessment. The participatory approach is

especially useful in the process of public consultations on a new legal act. It allows to understand the background (various stakeholdersâ&#x20AC;&#x2122; perspective) as well as to gain operational knowledge of the way of implementing an intervention.

Descriptive research design Its logic involves describing phenomena, their scale, progress, changes and dynamism by gathering raw data or existing data as well their analysis. It can be used for describing a situation, conditions, changes which are made over time in a given research area. It allows for gaining knowledge of the background which is especially useful at the stage of analyzing the problem for the needs of RT and ex ante RIA.

The authors of the handbook recommend that a starting point to every regulatory impact research (RT, ex ante and ex post RIA) should be recreating an intervention theory and then selecting these research designs (more frequently more than one) which in the best way will allow to verify the developed theory of change. Simultaneously, it is necessary to bear in mind that a relevant selection of research designs and thus relevant research methods and techniques (see part three of the handbook) is the greatest challenge facing each research team.

2.2. Planning and conducting research for the impact assessment process Impact assessment is an assessment of a normative act and its assumptions before or after its passage in terms of functioning the law. The underlying assumption of the impact assessment approach concerning legal regulations which is recommended in this handbook is based on the assumption that impact assessment should be based on evidence as far as possible. Compelling evidence is provided by social and economic research. Hence evaluation experiences10 with regard to designing and conducting research on the assessment of public interventions (of financial character) could be effectively used for progressing the process of assessing regulatory intervention impact. 10

eaders willing to familiarize with the process of contracting and collecting evaluation research R are recommended to read the guide: "Ewaluacja. Poradnik dla pracownikĂłw administracji publicznej", ed. S. Bienias, P. Strzemboszewski, O. OpaĹ&#x201A;ka, Ministry of Regional Development, Warsaw 2012. Experiences included in the Guide of the Ministry have become an inspiration for many elements contained in this handbook.

Evaluation can be defined as social and economic research conducted with the aim of assessing the quality and effects of undertaken initiatives. Source: Bienias et al., 2012.

An ideal solution would be to include all informative needs of individuals involved in the impact assessment process in one piece of research for recreating and verifying an intervention logic â&#x20AC;&#x201C;a designed or implemented solution to a social and economic problem (or various problem-solving options). However, research can also provide knowledge only for some informative needs which rise during the impact assessment process. In each case it is necessary to design and conduct the research properly in order to gain a reliable knowledge which is useful for answering questions related to IA. While designing and conducting research for the assessment of a legal act it is necessary to answer a basic question regarding a way of conducting research for the IA needs: Will the research be conducted internally by an analytical/research panel independent from the panel preparing a legal/legislative act? Will the research be conducted (outsourced) by an external panel at request of a contracting authority (office)? Although conducting the research internally is justified for RT and RIA (ex ante), in the case of assessment of functioning a law ex post it is recommended to outsource the assessment (evaluation). Outsourcing the evaluation is a guarantee of independence of the assessment process. A description of detailed conditions related to outsourcing research can be found in Appendix 1. It is important to make a decision about which research mode to adopt: participatory research or traditional one. As for the participatory model, a research panel accompanies officials preparing a legal act from the very beginning of the designing process. When it comes to a traditional approach, a ready-made act is a subject of research (at the assumption stage of RT or ex ante RIA). In view of dynamism of the process of preparing a legal act, this handbook recommends a participatory approach to conducting IA at the stages of RT and ex ante RIA. In this case the research is conducted alongside with the process of preparing a legislative act (the so-called participatory evaluation/ assessment).

Participatory model-additional conditions Selecting a participatory model for IA at the RT and RIA in the situation when the contractor is both an internal research panel and an external entity may bring additional benefits. Thanks to it researchers have a greater impact on increasing the quality of the document-preparing process. The researchers cooperate with individuals responsible for preparing the project – they assess, give opinions on particular elements of the document and play an advisory role. Recommended possible changes are introduced promptly, not after completing the whole process of designing a project. Close cooperation makes it easier for researchers to understand the assumptions of a legal act and allows for leading a constructive dialogue and achieving a consensus as for how recommendations are taken into consideration. As for a selection of a participatory model, this process should be initiated the moment initial assumptions concerning a legislative act in progress have been developed. An essential added value in the case of selecting a participatory model is a possibility of introducing strategic changes. These changes are easier to be made at an early stage of preparing a document when its general concept is being developed i.e. at the RT stage.

Within every impact research (evaluation or RIA) the following basic stages of the research process can be distinguished (Bienias et al. 2012, p. 37): • designing the research • conducting the research • using the research results The designing process includes the following elements: a) identifying and justifying a research topic and indicating the research recipients b) defining objectives, questions and research criteria c) defining the research scope d) defining research methods techniques e) defining an initial schedule of conducting the research and necessary resources In the process of conducting the research the following can be distinguished:

a) a stage of structuring the research b) a stage of collecting data c) a stage of analyzing and drawing conclusions The subsequent steps related to designing and conducting the research are described below. The matters referring to using the research results are described in the next section.

2.2.1. Structuring a research problem

We identify a topic and justify a decision about conducting the research As for impact assessment the research must be conducted pursuant to legal regulations which define RT, RIA and RIA ex post as obligatory. Depending on a kind of IA a research topics may vary. RT and RIA will focus on the analysis of the problem and the assessment of various options for solving a given problem whereas ex post RIA will mostly refer to the impact assessment of an implemented regulation. In other words, the research for RT and RIA will concern recreating proposed logical models or a selected solution to a problem and their verifying, including the cost/benefit analysis related to a selection of one of them. The research for ex post RIA, in turn, will mostly refer to the assessment of effectiveness and efficiency of the implemented solution. It is essential to have a clear vision of how the research results will be used and to engage key stakeholders in the process (e.g. by establishing a group supervising the research).

PAY ATTENTION! The research utility which means the extent of using the conclusions and recommendations is determined by the following elements: â&#x20AC;˘ engaging result stakeholders (main research recipients) in the research process: from the very beginning of the process they should play an important opinion-forming and informative role (as a source of data on a legal act in progress) â&#x20AC;˘ clarity of conclusions and recommendations and their relevant addressing â&#x20AC;˘ adopting a relevant way of publicizing the research results.

We define research objectives, ask research questions and define criteria for them The main research objective should unambiguously indicate the reasons for conducting the research and it should be expressed in short (max. 1-2 senten-

45 ces). Defining the main objective and possibly auxiliary specific objectives of the research is the most significant element of the research design or of the description of the subject of the contract (depending on the mode of conducting the research). The whole research process will have just refer to the defined objectives. The objectives indicate whether we want to investigate the impact, mechanisms or a way of implementing legal regulations. Depending on a kind of IA the research objectives will vary. Examples of the research objectives for ex ante RIA are presented in the box below.

Example 3. Research objective for RIA (ex ante) â&#x20AC;&#x201C; assessment of a ready-made legal act The main research objective is to assess a potential social and economic impact of introducing a law with reference to a scenario in favour of introducing no legislative changes. Moreover, the research also includes conducting an assessment of internal and external cohesion of a legal act and a designed system of its implementation.

How to correctly formulate research questions? After defining the main research objective and possibly specific objectives the most important stage of the designing process is formulating research questions. They clarify research objectives and are usually attributed just to the particular specific objectives. While formulating the questions, it is necessary to remember that they should concern specific informative needs related to the implementation of a legal act. The questions, or in a broader sense the whole research, are aimed not only at gaining a new knowledge but also using it for improving the quality of a legal act in progress.

Example 4. Research questions for ex ante RIA

The research conducted for ex ante RIA can answer especially the question: What will the social and economic impact (positive and negative) of introducing a designed legislative act be like as compared to the other defined scenarios (including, among others, a scenario of discontinuing all legislative initiatives)?

Secondly, IA can take advantage of evaluation experiences and make an ex ante assessment of: • internal cohesion of a document – defining to what extent the objectives (general and specific) are likely to be achieved by undertaking the defined initiatives; • external cohesion of a normative act with other regulatory documents as well as implemented financial interventions; • designed system of implementing a normative act in its organizational aspect. This aspect is to contribute to developing a proper system of monitoring and evaluation.

What criteria will we adopt in our research? In the evaluation practice an assessment is made on the basis of clearly defined criteria (most frequently such as relevance, cohesion, effectiveness, utility and sustainability) and questions or research areas which are adjusted to them. The boxes below present the application of evaluation criteria to RT, RIA (ex ante) and RIA ex post.

Example 5. Evaluation research criteria for RIA (ex ante) The research conducted for RIA ex ante can take advantage of a set of criteria used in the process of public policy evaluation. Ex ante evaluations are most frequently based on the four criteria (Compare: Bienias et al. , 2012, p. 29): • Relevance – analyzing the adequacy of prepared interventions in terms of identified social and economic needs/problems/challenged • Effectiveness – assessing the feasibility of designed objectives (and minimizing side-effects) with the use of available resources • Efficiency – assessing the preparation of documents in view of an optimal way of obtaining the expected results in relations to incurred expenditures • Cohesion (external and internal) – assessing designed documents in view of internal contradictions as well as contradictions between our project and higher-level or equivalent documents.

Example 6. Evaluation questions and criteria for RT Regulatory test referring among others to assessing the relevance of the diagnosis requires a description of the problem and its background, which means answering the following questions: • What does the problem involve? • What are the causes of the problem? • What is the scale of the problem? • What are the effects and impact of the problem (what may they be)? Another kind of knowledge gained with regard to a regulatory test is the knowledge of expected effects i.e. the assessment of expected effectiveness and efficiency. It is gained through answers to the questions: • How has the problem been solved in other countries? • What other solutions have been analyzed? (including non-legislative solutions) • What effects have been expected from particular solutions and why has the proposed solution been selected whereas the others have been rejected?

It seems that the research conducted for preparing Regulatory Test should particularly focus just on a reliable assessment of potential social and economic effects of the scenarios for solving the problem.

Example 7. Evaluation criteria for ex post RIA As for ex post RIA the main criteria of the assessment of functioning a law refer to efficiency, effectiveness and utility. Effectiveness is related to the question about the extent to which the objectives have been achieved. These objectives are to be achieved thanks to a law put into force (particularly according to the indicators defined in RIA/RT). In view of the above, effectiveness is related to a question about the desirability of putting the law in force:

Have the problems which were the cause of putting the law in force been really solved? Efficiency refers to obtaining beneficial effects in relation to the costs of putting the law in force (and its observance). An additional assessment criterion should be utility which identifies and assesses all the criteria, including unexpected (beneficial and non-beneficial) effects of the regulation.

We define the research scope With reference to most research we define the scope of conducting it in at least three aspects considering the following (Bienias at al. 2012, p. 44): • a subject – we can focus on the whole draft of a legislative act in all aspects or a selected part solving a selected social and economic problem (more information on it – see the section concerning intervention theories) • a territory – the research can include a country as a whole or a properly selected sample • time – over what time we investigate the addressees of our regulatory intervention.

2.2.2. Defining a way of conducting research

We define research methods and techniques While preparing a way of conducting the research we are able to define in detail research methods and techniques on our own or we can leave clarifying a research methodology to a research contractor. A client defines at least minimum methodology standards which can include selected research designs (particularly to be used in the case of outsourced research). As for the former approach, we clarify a few essential things, among others, we describe in detail the methodology necessary for conducting the research. A detailed approach to defining research methods and techniques is described in section 3. Examples of research approaches are also presented in the case studies. We define a research budget Assessing the research costs is one of the most difficult tasks facing a panel which designs the research for impact assessment. The basic knowledge ne-

cessary for assessing the research budget concerns methods and techniques which will be used in the designed research. Additionally, in the case of ex ante research, the necessary knowledge refers to a number of stages of consulting a draft legal act. As for the research designed to be conducted by internal panels it is possible to send a few research companies an enquiry, presenting them the main objectives which are to be achieved by the impact assessment based on evaluation approach. The replies will provide information on both the costs and feasible methodology (the more information we send to potential contractors, the more detailed calculation of costs we will obtain). The research budget can be also defined by comparing the costs of similar projects previously implemented. However, it is necessary to pay much attention to the similarities, i.e. the subject of the researched legal act, the methods and techniques used or a kind of assessment (ex ante research is not a good model for ex post research). While designing the research budget, it is necessary to also include organizational costs (operation of the office involved, costs related to organizing meetings or translating materials). When it comes to both outsourced research and research conducted internally within the office, one way of calculating the costs is taking into consideration a number of man-days referring to the work of experts conducting the assessment. If the research is outsourced, the time can be converted into financial means. In the case of conducting research internally within the office, a calculation will include the â&#x20AC;&#x2DC;opportunityâ&#x20AC;&#x2122; cost (at this time officials could fulfill their other duties). We define a research schedule While designing research we must guarantee sufficiently long timing for conducting it and divide it properly into stages and tasks. The period of conducting the research will be very much differentiated and it will depend on the research scope and complexity of a given legal act. In the course of impact assessment based on evaluation approach, the Legislative Panel along with the internal research/analytical panel prepares a joint schedule of conducting the research. An example of schedule model can be found in Appendix 2. If the research is outsourced it is necessary to prepare two schedules. One prepared by the client should have a broader scope which

includes not only the course of the research itself but also all initiatives and activities to be undertaken before and after conducting relevant assessment. The other schedule should be developed by the contractor and it should concern the research process itself. The schedule should be presented for the first time in the offer. Its final version should be presented in the methodological report.

2.2.3. Conducting the research

In the case of outsourced research, the design research process ends with starting the procedure of outsourcing the service (research) within Public Procurement Law. A description of detailed conditions related to outsourcing research can be found in Appendix 1. The process of conducting the research includes the following: a) a stage of structuring the research b) a stage of collecting data c) a stage of analyzing and drawing conclusions A stage of structuring The process of procurement procedures with reference to conducting the evaluation research ends with a selection of the contractor and signing a contract. Also, at this stage the first working meeting is held in which the client can clarify its expectations. At the beginning of the research all members of the research panel should sign a declaration of confidentiality (an example of the declaration can be found in Appendix 2). The client also provides the contractor with a letter of credence, whose model can be found in Appendix 2. The first research product is a draft of methodological report. It is also worth preparing a methodological report presenting a research concept and a way of conducting it (methodology and a way of organizing the research) in the case of the research conducted internally. The methodological draft report should be discussed at the meeting in which the research panel (internal or external) presents methodological assumptions and the legislative panel or the panel commissioning the research have a possibility of making their own remarks on the proposed methodology. During such a meeting the research concept and designed researched methods and techniques are discussed in detail. At this stage it is necessary to define availability of the data essential for conducting the research. It is also important to gather informa-

tion about individuals who can support conducting the research and provide other relevant documents and data. Thanks to cooperation and partnership of both the clients and the contractors as well as other partners (research stakeholders) from the very beginning of the research their demand for the assessment results is generated. A stage of collecting data After the methodological report has been agreed on, the process of collecting data by researchers starts. It concerns collecting secondary data available at different data bases such as statistical data, those included in governmental documents and scientific publications as well as raw data. At this stage special attention should be paid to the timeliness of collected data as well as their reliability/credibility. Section 3 of the handbook presents in detail methods and techniques useful at the stage of collecting data and indicates the sources of secondary data â&#x20AC;&#x201C; databases useful at this research stage. Practical examples of the ways of conducting the research at the stage of data collecting are presented in point 4 of this section which includes a presentation of methodological assumptions of an example of research on laws using an evaluation approach. A stage of analyzing and drawing conclusions The collected data are a basis for analyzing and drawing conclusions, i.e. searching for answers to research questions as well as assessing the collected material in view of adopted assessment criteria. At the stage of analysis special research methods and techniques are used to assess the collected material and draw conclusions and recommendations. A detailed description of the methods and techniques useful at this stage of analyzing and assessing the collected data can be found in section 3 of the handbook.

2.3. Strategy for the use of research results in the IA process

Experiences regarding the evaluation of projects and programmes co-financed with the resources within Cohesion Policy indicate that the results of the best research are useless if not used. Therefore bearing in mind using the results of the research for IA it is advisable to develop a strategy for a use of the results of conducted research and analyses.

A strategy is nothing but a design of initiatives â&#x20AC;&#x201C; a step by step description how to achieve a given goal. A strategy also takes into consideration all factors which foster and impede to achieve objectives. So, a strategy is a road leading to the objective. It is worth at least thinking over such a strategy for using the research results but an optimal solution would be to put it in writing. If a use of the research results depends on the research utility, designing how to use these results should begin along with starting a design of the whole research. It will allow, among others, for proper designing a structure of the research report. At this stage it is also advisable to develop a strategy for using the research results and it should be created by individuals who either conduct the research on their own or prepare outsourced procurement. No matter whether the research is conducted for the needs of RT, RIA and RIA ex post, the process of preparing a strategy for using their results needs answers to a few basic questions which are presented in the diagram below. Diagram 4. Strategy structure of communicating research results

What do we want to communicate?

What has been changed as a result of communicating information?

Strategy for communicating research results

What language do we use?

Source: by authors

Whom do we want to inform?

How do we communicate information?

53 2.3.1. Identifying key information Research is associated with multi-page reports often written in an incomprehensible scientific language. It makes it difficult to gain knowledge from the results of the conducted research. Going through thick reports to find the information which is essential for drafting a legal act and its IA is really laborious. Let us think about decision-makers who in order to make a decision would be forced to read a few of such reports. Obviously, the day would be too short for this. Therefore, while designing a strategy for using the research results it is necessary just at the very beginning to ask what kinds of information are essential at different IA stages. Informative needs in the RT and RIA (ex ante) process differ from the needs at the stage of RIA ex post. In the course of drafting a legal act, the key information refers to different solutions proposed and the costs and benefits of their implementing. At the stage of assessing the functioning of a law, the conclusions on the effects of intervention and recommendations for improving the law functioning in the future are crucial. Key information, i.e. the research results as well as conclusions and recommendations should be included in a report on the research which is a final product of the research. The scope and form of communicating the key information depend on informative needs of different groups of recipients.

2.3.2. Defining target groups for communicating information

The next step to be made as for a strategy for the use of the research results is to define recipients of the information communicated by us. It may be necessary to communicate different elements of the research results and present them at a different level of detail and in a different form to different groups. The main groups of recipients of the research results in the impact assess-ment process are as follows: • decision-makers • individuals preparing a given legal act /legislative panel • entities which are subject to a given legal act • participants of consultation • media and society as a whole Preparing a strategy for using the IA research results, it is necessary to bear mostly in mind two groups of recipients:

• decision-makers – the research results should provide evidence which is necessary for making decisions on a given policy or an issue regulated by the act. • Individuals preparing a given legal act – they obtain the information on the background, mechanisms and effects of the planned interventions which could be used in the process of developing the assumptions of the legal act, the draft legal act as well as for preparing the amendment of the legal act. As for these two groups of recipients, it is necessary to remember that in order to increase a sense of responsibility for the research results, the groups should be informed about the research progress, or even included in its designing. It will bring a positive effect on using the research results. When the prepared legal act is more and more in progress, entities which are subject to this act and participants of public consultations should be also included into the group of the recipients of the research results concerning this act. As for the research ex ante and ex post on a legal act, it is also necessary to bear in mind media and society as well as bodies supervising the enforcement of a given legal act. The information communicated to these groups is mainly to prove transparency of initiatives undertaken with the aim of preparing the legal act and show that the decisions on the form of legislative intervention (prepared or amended) are based on evidence (the results of the research conducted). It is worth pointing out again that depending on different recipient groups, the communicated messages should be differentiated.

2.3.3. Selection of proper tools and a form of communicating the research results

Preparing information on the research results, we have at our disposal a wide range of communication tools to be used both when the research is being conducted and after it is completed. The table below shows what tools can be useful as for specific target groups.



  

Participants of consultations Media Society as a whole



Individuals preparing IA Entities which are subject to a given legal act



Discussion

Individuals preparing a legal act/Legislative Panel

Information leaflets



Presentation of research results



Final report

Executive summary

Decision-makers

Press information

Note

Table 3. Examples of tools of communication for specific recipient groups

 



 



Source: by authors

Designing outsourced research, it is possible to let external experts develop some materials (an informative leaflet or a presentation). A note should be short (1-1,5 pages) (decision-makers rarely have time to read longer texts). It should include the most important information, particularly synthetical conclusions and recommendations on the research. It should be written in a possibly comprehensible language. Decision-makers can make a decision on further action on the basis of the note. Also, when a given individual is especially interested in one issue they will have a possibility of using a full report. An executive summary is a brief summary of the report prepared mainly in view of recipients who cannot read the whole report. The executive summary should not exceed 10 per cent of the whole report text and it should focus on

conclusions and recommendations resulting from the report presented in easily understood words. Such an executive summary should not include descriptions of methodology, footnotes, graphs and tables. A well-prepared summary contains headings, bullet points, links to specific parts of the report (e.g. page numbers). The links will make it easier to find justification for a given conclusion of a special interest. Because the executive summary encompasses the core of the research report, including conclusions and recommendations, it is a document addressed mainly to decision-makers but also to media. Press release is a short document addressed to the press, on the basis of which a journalist can prepare an article. The note should contain a title or subtitles, a lead (the first paragraph following the title summarizing the most important information in 2-3 sentences) and a body (subsequent paragraphs). The text can include some information about the individual dealing with a given matter and their business contact details. The text should be written in the third person singular in a way which avoids all value judgements and expressions and which eliminates unnecessary adjectives and adverbs. An advantage of a good press release is short sentences written in a language comprehensible for an average reader. The most important information (answering who/what/ where/when/why questions) should be placed at the beginning of the text (KomendziĹ&#x201E;ska, available on: http://notkaprasowa.pl/poradnik.htlm). Final report is theoretically the most important product of the research. It should be remembered that it is a variation of scientific report - and as such â&#x20AC;&#x201C; has its own demands. The final report is a summary of the research results. The ultimate success of the initiative depends to a large extent on whether the report contents is presented clearly and transparently. The basic requirements concerning the structure of the final report are presented in the Appendix 2. The report part referring to methodology and data analysis must be written in a scientific language. However, the sections containing conclusions and recommendations should be presented comprehensively in short sentences and commonly known words. It is also advisable to use frequently enumerations and bullets. What should be avoided is scientific terms, sequences of nouns as well as lengthy multi-word forms (Maziarz et al., 2012, p. 55). The report is addressed to individuals conducting IA and those involved in drafting and enforcing a legal act. Also, recommendation addressees, decision-makers, media,

experts and academics may show interest in the report. However, the conclusions should be communicated to each group in a separate form which matches their specificity (described in this subsection). Presentation of the research results usually has a form of an oral presentation illustrated by a multimedia presentation. Depending on to whom it is addressed (citizens or representatives of other administration units) we must remember about the appropriate language of our speech. The less professionally the group is connected with the presented subject the simpler language should be. It is necessary to avoid a specialist language. Also, while preparing a multimedia presentation we should stick to the following rules:

• Do not write on a slide what you are going to say because the presentation will be just boring • Do not overload the slides with a text or other things –sometimes less means more- the more the text and illustrations on a slide the fewer details will be noticed by listeners • Illustrate what you are talking about with simple graphs, tables, diagrams • Speak at first, then show single slogans to be remembered; in this way listeners remember what they have just heard • Use one typeface, three styles or sizes; too many different typefaces distracts listeners from what has been written (how it has been written) • Use margin guides for a symmetrical layout of the text and other things on the slides; it ensures a clear structure of the slides and a sense of order

Source: by authors, based on Kluj, available on: http:/www.instytut.com.pl./sites/default/files/Jak pisac strategie komonikacji-przewodnik.pdf

While making a presentation it is worth keeping eye-contact with the listeners. We should not read from a paper sheet (actually we can glance at a note with the main points of the presentation). It is advisable to speak clearly and not too fast (but also not too slowly as the listeners will stop being interested), modulate the voice, change the pace and the voice timbre, stress some sentences, make pauses, ask questions, also to the listeners, provide examples and anecdotes in order to attract the listeners’ attention. Informative leaflet consists of a few pages of informative text about the research and is usually addressed to a wide group of recipients. Reading the leaflet should take as little time as possible and at the same time it should provide

as much information as possible. The information included should be substantive and specific, so defining the role of the leaflet should be precisely defined. Its layout often contains subtitles, subheadings and bullet points which ensure conciseness and transparency. The leaflet text can be illustrated by graphs, tables and diagrams. The leaflet should pass on concise information in short sentences, preferably in ordinate elliptical clauses. The language must be simple and comprehensible. Additionally, it is worth remembering that the leaflet should be aesthetic and visually attractive. If we consider a discussion, it is good thinking over the issues which could be touched or the questions to be answered. Discussion could be mostly a good communication tool if we want to discuss the research results, conclusions and recommendations with its recipients. It is also a good tool in the case of controversial subjects or if we want to make a decision based on the research results at the presence of a wider group of stakeholders. In view of the above it would be a good idea to design the discussion progress. To make it more effective it is advisable to follow a few basic rules:

• Listen to others and they will listen to you • Never shout down other speakers and never cut into what they say • Do not lose your temper; the speaker may provoke you intentionally in order to take advantage of your momentary weakness • Do not be afraid of stating your viewpoint; if you do not present your point of view, it will never be taken into consideration • Speak to the point; too many digressions and departing from the subject not only confuse your message but also can irritate other interlocutor • Change the form of speech depending on the listener • Do not try to refute your interlocutor’s opinion by pointing at their drawbacks; criticize arguments, not people • Do not refer to prejudicial stereotypes • Do not try to convince anybody at any price; respect other’ viewpoints • Use examples illustrating a given subject. Visual images are more appealing than abstract ideas

Source: https://edukacjamedialna.edu.pl/lekcje/sztuka-dyskusji

Discussion is run using the research materials prepared earlier. So as to obtain the effect acceptable by possibly the biggest group of interlocutors, the discussion should be run by an experienced but disinterested moderator. At the end of it the most important issues which were agreed on or touched should be summed up and written down in a memo.

2.3.4. Monitoring the use of research results

Publicizing the research results is a necessary condition for the use of research conclusions and recommendations, and in consequence for the quality of a draft legal regulation. In order to ensure implementing recommendations in the case of outsourced research, it is important that the research panel should cooperate in this respect with the client because the implementation progress depends also on the client situation. One way of ensuring close cooperation between the research panel and the client is using participatory approach as a research method. In this model the research panel, conducting impact assessment and cooperating directly with the Legislative Panel, should take care of the research conclusions and recommendations to be implemented continuously in the whole research process which is simultaneously in progress with the process of preparing a draft legislative act (RT and RIA). Referring to good examples of using conclusions and recommendations within the system of Cohesion Policy evaluation in Poland and having in mind the IA needs it is possible to adopt a table of recommendations used for evaluation research. An example of such a table is presented below:

Recommendation

Way of implementing

Justification

Extension of modular project bank and â&#x20AC;&#x201C; implementing multiannual projects (pp. 17,19, 24-25, 27)

An existing module of biannual projects should be especially extended with reference to the priority of development of small and medium-sized enterprises (SME) so that it will be possible to adopt a larger number of projects. In view of this action, long-lasting support is exceptionally essential. It is worth considering to develop SME priority only through biannual modular projects.

Initiatives addressed to SME are much more difficult to be implemented and pose a higher risk than, for instance, social projects. To obtain the desirable effects, initiatives need to be undertaken over time. Most sustainable are complex forms of support whose specific actions (project promotion, study visits, trainings increasing competence) must be undertaken in a proper order. It influences lengthening the project cycle. Therefore, considering even small project budgets, it is advisable to take account of adopting initiatives undertaken in a time perspective longer than one year.

Addressee Ministry for Foreign Affairs

Source: IDEA for Development Foundation

Such a table of recommendations should be discussed with the recommendation addressees as they possess the greatest knowledge of the proposed solution and its implementation feasibility. The stage of consulting recommendations is finished when the table is approved. It is when the process of their implementation starts and when it is necessary to monitor the progress. Monitoring the research results allows mostly for assessing whether the financial means spent on the research have been efficiently used. Seeing how many research recommendations have been implemented, we are able to assess whether the research has been utilitarian. How to monitor the implementation of recommendations? In this respect, the system of Cohesion Policy evaluation is very helpful again. Good practices have already been developed in the system. The National Assessment Unit (NAU) in Poland introduced in July 2008 the Evaluation Research Conclusion and Recommendation Managing System. The system makes it possible to monitor conducting and managing the implemented recommendations. Once the research has been completed , the responsible body passes on the table of recommendations to the NAU and then it is imported to the main system. Thanks to it, further monitoring of the use of research results is possible.

At the same time, the research results concerning one legal act can be used in the process of drafting other legal acts or even developing whole policies. Knowledge produced for a given IA may be reused i.e. within the systematic review of all research conducted in a selected public policy area. It is conditioned upon gathering (and making public) all results of research conducted for the purposes of IA. It is proposed thus to create a data base of research conducted for IA and its integration with other databases. It would make it possible to use different analytical works for the conduct of the debate based on evidence. Among the databases, which should be integrated, a database of evaluation reports must be mentioned (the Polish one is available at: www.ewaluacja.gov.pl)

2.4. Examples of EX ANTE and EX POST assessment of a law conducted within the project

Within the MORE project it was planned to conduct some examples of the research on legal acts at the stage of drafting them (before a law is put in force) and at the ex post stage (after a law entered into force). They are practical illustrations of how evaluation tools can be used for conducting impact assessment The descriptions of the selected examples presented below show how the research logic was designed and how the research was conducted (following the methodological reports prepared for the research11). Example 1. Ex ante assessment of the Law on amending the Law on security of food and nutrition of 25th August, 200612 The subject of the research is the Law on amending the Law on security of food and nutrition of 25th August, 2006 (Journal of Laws of 2014, items: 1662, 1722). One of the introduced amendments to the regulation is a ban on fast food at schools. The research objective is to assess the law consequences in view of designed intervention effects, side-effects and the mechanism of change. 11

t the moment the handbook is being prepared no research results are available because the reseA arch is due to start on 1st June 2015. In view of this the presented examples have been based on the methodological reports developed for the research. he example is based on: B. Lendzion et al. (the methodological report prepared within the MORE T project).

Logical model This law takes advantage of the logic of influencing people behavior which is based on prohibition. IF

THEN

we introduce a ban on selling fast food in school shops and on advertising such food

children and youths will eat healthier food and their nutrition conscious-ness will increase

AND THANKS TO IT We will prevent the obesity epidemic among youngsters, which is a severe public health hazard

Source: B. Lendzion et al. (methodological report developed within MORE project)

If simplified, the logic of the analyzed intervention follows the progression: Research designs and research tools To verify the adopted logic of the regulation the following research designs can be used: • Meta-analysis (systematic review) – it is to provide knowledge of the mechanisms applied at schools within other programmes/in other countries in order to change children’s eating habits and decrease obesity in children as well knowledge of the effects of public initiatives in this area. • Quasi experiment – it will be used to obtain the knowledge of actual effects of putting the regulation in force, which in fact, is answering the question: will putting the regulation in force decrease fast food consumption, differentiate pupils’ diets and change their approach to eating habits? For the research need the changes made at the schools which are subject to the ban on selling fast food will be compared to the changes typical of the schools where the regulation is not applicable. • Theory-based evaluation- it will be used in order to obtain knowledge of the mechanism of change which was put into operation due to the law and especially of children’s behavior and changes made in their schools and families resulting from the law enforcement. • Case study – it will be conducted at schools which voluntarily have introduced changes influencing eating habits. The case study will make it possible to observe mechanisms of change and effects of such initiatives

as well as factors determining their success or failure. Within this research design the existing data as well as raw data from field studies will be used. â&#x20AC;˘ Participatory approach â&#x20AC;&#x201C; it helps discover motivation and opinions of the law stakeholders, thanks to which it will be possible to obtain knowledge of the intervention recipients and the background for enforcing the law. Participatory approach is also useful in the process of conducting social consultations on draft legal acts. The use of this research design makes it possible to obtain answers to the questions about the stakeholdersâ&#x20AC;&#x2122; behaviours, decisions and attitudes.

Example 2. Ex post assessment of the Law of 14th March, 2014 on amending the Law on promoting employment and on labour market institutions and some other laws13 Research scope The subject of the research is the Law of 14th March, 2014 on amending the Law on promoting employment and on labour market institutions and some other laws. The amendment objective was to increase the impact of the labour market policy on the labour market situation through a number of initiatives in view of improving the efficiency of labour offices, the quality of their services as well as introducing new tools supporting the unemployed. Due to a wide range of changes generated by the amendment, the research scope was limited to assessing the following key elements of the reform which refer to a) a change in the algorithm for dividing the Labour Fund, b) introducing a mechanism of rewarding labour offices for the effects of their work, c) profiling the unemployed, d) ordering activating initiatives, e) increasing the flexibility of the labour office work14 . Logical model Due to the synergism of the changes generated by the law it is necessary in the first place to recreate the intervention logic i.e. to make an attempt of iden13 14

The example is based on the methodological report prepared within the MORE project Ibidem, p. 3.

tifying the key assumptions, designed initiatives and expected effects. Then the intervention logic will be verified thanks to researching the impact of the reform key elements on the way in which labour offices work. It is assumed that the key changes generated by the reform are related to the synergism of mechanisms of rewarding for effects, which means that labour offices are rewarded depending on achieving specific efficiency indicators or the quality of work. The research will focus on the mechanisms referring to New Public Management (NPM). The research objective is to verify the adaptation processes in offices in view of improving their work which were generated by the introduced mechanisms. The hypotheses included in the research referred to the impact of the mechanism of rewarding district labour offices for their effects as well as of the mechanism of ordering activating actions on an increase in effectiveness of support for the unemployed returning to labour market. Research design and research tools The following research designs will be useful for verifying the adopted hypotheses: â&#x20AC;˘ Meta-analysis (systematic review) will be useful for the synthesis of the previous research results on using the NPM-based approach in public administration, particularly in a view of mechanisms of rewarding for effects â&#x20AC;˘ Theory-based evaluation will serve for recreating the intervention logic (the mechanisms introduced by the reform) as well as for searching evidence for the expected changes to be made in reality (improving the support effectiveness as for the unemployed returning to labour market). Within this design the following research tools can be used: 1) logframe matrix for recreating the intervention logic and mapping, 2) individual interviews with employees of district labour offices (DLO), career counsellors, representatives of the labour market institutions from the outside of administration, labour market experts, 3) analysis of existent documents, 4) expert workshop (experts and DLO employees), 5) CAWI analysis, 6) DLO case study (there will be 5 case studies presented in the subsequent research design).

â&#x20AC;˘ Case study â&#x20AC;&#x201C; within the research 5 case studies will be conducted for an in-depth analysis of the situation in the researched district labour offices. The following research tools will be used: analysis of the existent documents and individual interviews with DLO employees and career counselors.

PART THREE RESEARCH METHODOLOGY

3.1. Methods and techniques of collecting and analyzing data Knowledge of methods and techniques of collecting data plays a key role at different stages of the regulatory impact assessment process. Taking advantage of the methodological acquis of evaluation research, which refers to social sciences in a broad sense, can be helpful at various stages ranging from identifying a problem to be solved, developing different feasible solutions, assessing different options in order to choose one of them. Getting acquainted with the approaches presented below is also essential in view of assessing and evaluating the impact of putting in force a regulation. Individuals involved in IA should use both quantitative and qualitative methods. It is the basic division of methods in social research. As E. Babbie states: â&#x20AC;&#x2122;Distinction between qualitative and quantative data basically is in line with distinction between numeric and non-numeric dataâ&#x20AC;&#x2122; (Babbie, 2007). Obviously, while using the two kinds of methods, we must take account of some limitations ascribed to them. Thanks to quantitative methods, we obtain figures which are then analyzed by using statistical tools. If a research sample is relevantly defined, the results can be also reliable with reference to the whole population. Qualitative methods, in turn, allow for in-depth observation. The results are more precise, however they cannot be (in most cases) generalized . A selection of method will depend on what we want to find out. The optimal solution is to combine both methods within the research.

The table below presents the most important differences between the two approaches:

Table 4. Qualitative and quantitative methods – different approaches Quantitative methods

Qualitative methods

Used when we want to find out ‘how much/many’

Used when we want to find out ‘why’?

We obtain numeric data (figures)

We get acquainted with the process, we interpret what has occurred

Data are easily compared

It is difficult to compare data

More objective

Risk of subjectivity

Responder is inactive

Responder plays an important, active role

Large samples

Small samples

Outer researcher’s perspective, limited contact with data sources

Researcher is inside the process, direct contact with data sources

Source: by authors based on: Nikodemska-Wołowik, 1999.

In order to select the proper set of methods and techniques we can use research designs (described in part 2 of this handbook). In this situation we firstly point at the research design/designs we want to adopt and then we select the proper set of research methods and techniques. It is also necessary to bear in mind that some methods and techniques described in this handbook can be used not only for developing IA. The research which is properly designed can provide many pieces of valuable information to be used in the social consultation process. This section presents the most popular methods and techniques (both qualitative and quantitative) of collecting and analyzing data which can be essential in view of preparing IA. Additional emphasis was put on selected methods and techniques especially useful for analyses and drawing conclusions. The description of the methods is proceeded by the information on potential data sources which could be referred to in the process of collecting information necessary for impact assessment.

69 15

While drafting good quality legal acts, it is also worth taking account of other trends which are predominant in social research. One of them is behavioural analyses casting completely new light on the way in which people make decisions by referring to the psychological basis of their behavior. Behavioural perspective allows to understand better motivation of an individual to make a decision. It may be useful in view of conducting impact assessment because, as it is indicated by Olejniczak and Śliwowski: ‘the better the creators of interventions adjust their form and logic to the ways of making decisions by citizens, the better effects will be brought by initiatives undertaken by public authorites’ (Olejniczak and Śliwowski, 2014). Adopting behavioural analyses to impact assessment can be the next step following the experiences of public intervention evaluation.

3.2. Data sources-catalogue of useful databases and sources (national and foreign) Apart from a knowledge of methods and techniques, essential factors are reliability and timeliness of the data we will use in the IA process. In terms of data origin, we can reach for raw data, specially collected in order to response to the problem, as well as secondary data which derive from public statistics, governmental documents (strategies, expert reports, and other reports and legal acts), scientific publications and others. Below in the text are mentioned some sources which can be useful while searching for secondary data, whereas section 3.3. presents methods and techniques partially useful for collecting such data.

ATTENTION: while designing research, in the first place we should use available sources of secondary data, and only if there is lack of existent ones, it is necessary to reach for methods and techniques aimed at obtaining raw data.

ore on the topic: Olejniczak and Śliwowski, 2014; e.g. Shafir, 2013, as well as Journal of Behavioral M Decision Making - http://eu.wiley.com/WileyCDA/WileyTitle/productCd-BDM.html (accessible on 20th April, 2015.

70 3.2.1. Statistical data As for public statistics, firstly we should take advantage of the resources of the Central Statistical Office (CSO) (http://www.stat.gov.pl) providing access to basic statistical data and offering a set of tools which could be successfully used in the impact assessment process. The two public bases particularly useful in this respect are presented in the box below:

The development monitoring system STRATEG is a base of key indicators for monitoring the development of the country as a whole and of lower regional levels. It also contains basic indicators for EU member countries, which allows for data comparability. The STRATEG is updated at least every three months and it is a system accessible to the public. @address www:http://www.strateg.stat.gov.pl The Local Data Bank, which is administered by the CSO, is the largest public database concerning the demographic, environmental, social and economic situation at the level of voivodships, districts and municipalities as well as regions and subregions. @ address www:http//www.stat.gov.pl/bdl/html/indeks.html

The European Statistical Office (Eurostat) is responsible for developing statistical forecasts and analyses regarding the EU and EFTA aimed at fostering the decision-making process in the EU bodies. The Office is also in charge of monitoring and coordinating statistical offices of member states in view of unifying national statistics. The data collected by Eurostat make it easier to compare situations in various areas with their counterparts in other countries. As for some data, it is also possible to make comparisons at regional level (regional data regarding Poland have been accessible since 2004). @ address www: http://ec.europa.eu/eurostat

The necessary data can be also found in the resources of the World Bank (WB) (the website http://data.worldbank.org/ has a section containing statistical data) or the Organization for Economic Cooperation and Development (OECD). The access to the OECD libraries is, among others, through the OLIS system (http://www/oecd.int/olis) after prior authorization or through Common Internet Platform for central government officials16 (intranet.gov.pl). 16

The Platform is an element of the project Modernisation of the Management System and Information Flow in Central Government Administration cofinanced by the European Social Fund within priority V.

Thematic reports and analyses developed by the WB and OECD can be useful particularly in view of collecting in-depth data for comparisons at an international level.

3.2.2. Governmental documents

Searching for access to main strategic documents essential for conducting development policy in Poland, it is advisable to visit the website of the ministry managing regional development (http://www.mir.gov.pl) where links to specific strategies developed by the government can be found. Regional level data in Poland can be obtained from strategies of voivodship development which, as a rule, are accessible through the websides of the Marshal offices. At the voivodship level there are also regional territorial observatories (RTO) which monitor and evaluate development processes in the regions and the effects of the conducted policies. By publishing the research results and analyses, ROTs provide policy-makers with the data necessary for conducting development policies. It is worth stating that at the central level there is also National Territorial Observatory located in the Ministry of Infrastructure and Development, which plays a similar role. The problem can be also identified through conclusions drawn on the basis of evaluation research which are publicized, among others, on the website: www.ewaluacja.gov.pl. Moreover, the websites of particular ministries can contain useful analyses, expert reports, etc. Searching for up-to-date legal acts it is advisable to refer to the base of the Chancellery of the Sejm of the RP entitled Legal Act Internet System (LAIS) containing provisions of legal acts published in the Journal of Laws and Polish Monitor. @ address www:http://isap.sejm.gov.pl

Through http://legislacja.gov.pl it is possible to find the base of the Government Legislative Centre which contains draft legal acts, including drafts of law assumptions as well as developed ex post RIA. â&#x20AC;&#x2DC;Good Governanceâ&#x20AC;&#x2DC; under the Operational Programme Human Capital conducted by the Department of Strategic Analyses of the CPM and the Common Service Centre.

3.3. Characteristics of selected qualitative and quantitative research methods and techniques Below there is a description of research methods and techniques which can be useful at different stages of preparing impact assessment. They are in the following order of priority: • stage of defining the main assumptions and methodology of conducting research (structuring), • stage of collecting data to be used in research (observation), • stage of analyzing the data collected at the phase of observation (data analysis), • final assessment stage along with drawing conclusions and recommendations (assessment). The techniques are grouped according to the above stages although they can be also useful at other stages, depending on a specific situation to which the research refers. Ascribing particular techniques to the research stages is shown in a table at the end of this section. The presented techniques do not take advantage of all tools which can be used for impact assessment but focus on the most popular ones in public policy evaluation. SWOT/TOWS analysis SWOT analysis is a heuristic technique used at the stage of research structuring (if combined with TOWS, it is also excellent at the assessment stage). It allows for sorting out the obtained information. As a rule, it has a form of a table with four boxes containing information respectively on: strengths (S), weaknesses (W), opportunities (O) and threats (T) regarding the researched issue. In the first place the analysis focuses on an internal environment (S and W) and then it goes on to external factors (O and T). TOWS analysis reverses this process, i.e. it is conducted ‘from outside towards inside’. SWOT/TOWS analysis can be useful17 at the stage of both identifying a problem to be solved and designing a solution to it. SWOT can help obtain the answer to the question ‘why a given solution has not been applied’ in the part 17

escription of each method includes the paragraph concerning the recommended application of the D method at different stages of IA. The following points concern the elements of IA required in Polish IA system. Nevertheless, they should be taken into account in the process of IA regardless of the national context.

describing analysis of alternative solutions. Additionally, this technique can be also useful with reference to regulations related to enforcement of EU directives when we analyze flexibility while selecting particular solutions and analyze feasible solutions. Another SWOT application regards sorting out the information on solutions used in other countries in view of their utility in national conditions. With reference to ex post RIA SWOT/TOWS analysis can be useful in view of identifying the costs and benefits of functioning a law which are to be subject of further assessment. Summing up

What does SWOT/TOWS analysis involve? We use a tabular format to analyze strengths and weaknesses as well as opportunities and threats typical of a given system and by this we analyze both its inside and its environment. Strengths

Weaknesses

• It focuses on specific features of the researched issue

• There are possible discrepancies in interpreting

• It allows to identify the factors on which we have influence and the factors independent of us

• It is a simplified analysis – it uses general statements

• It is aimed at effective solutions to the problem Source materials on SWOT/TOWS analysis  Bończak-Kucharczyk E. [2008], Strategie rozwoju regionu i miast. Metodologia i wskazówki praktyczne, Minigo.  Obłój K. [2007], Strategia organizacji, PWE, Warsaw.  Dziemianowicz W., Szmigiel-Rawska K., Nowicka P., Dąbrowska A. [2012], Planowanie strategiczne. Poradnik dla pracowników administracji publicznej, Ministerstwo Rozwoju Regionalnego, Warsaw  Strahl D. [red.] [2006], Metody oceny rozwoju regionalnego, Wydawnictwo Akademii Ekonomicznej im. Oskara Langego we Wrocławiu, Wrocław.

While conducting SWOT analysis it is essential that we should: • use only precise, verifiable statements • continuously prioritize identified factors • while analyzing ‘challenges’ ask the questions: 1. whether the ‘strengths‘ will not contribute to creating ‘opportunities‘ 2. whether eliminating any of ‘weaknesses’ will not contribute to creating ‘opportunities‘ • remember that opportunities and threats are the factors which are outside the researched problem

74 3.3.1. Brainstorming It is a technique for finding creative solutions to problems which is useful at the structuring stage. It is based on team work of the expert panel engaged in researching an issue and coming up with ideas of solving a problem. The technique is relevant, among others, at the stage of developing working hypotheses, logical matrixes (see point 3.3.3) aimed at recreating the logic of analyzed initiatives/activities as well as at the stage of preparing specific research tools and of interpreting the effects brought by quantitative approaches. Brainstorming can be successfully used when the problem to be solved is being identified or when the solution is being developed. The technique may be also helpful in the analysis of alternative solutions and in debate on solutions applied in other countries. Brainstorming within ex post RIA can be also useful as for analyzing the relevance of putting a law in force. Summing up What does BRAINSTORMING involve? Brainstorming is a discussion among research panel aimed at collecting creative ideas of how to solve a problem. Strengths

Weaknesses

• it is possible to obtain a lot of interesting problem solutions

• it poses a risk of dominating the group by a strong individual and imposing their viewpoint on the others

• it does not generate high costs • it can be widely used for different analyses Source materials on BRAINSTORMING:  Góralski A. [1989], Twórcze rozwiązywanie zadań, PWN, Warsaw.  Martyniak Z. [1999], Metody organizacji i zarządzania, Cracov.

While arranging brainstorming it is essential that we should: • properly define the problem • be creative – the more ideas, the greater chance of solving the problem • moderate the discussion – only one person speaks at a time • well design the discussion – we define the limit of time or ideas; it is worth remembering that the bigger the group is, the more time is needed • avoid criticizing other participants’ ideas • discuss in detail the most promising ideas among those included in a ready-made list 3.3.2. Mindmapping Mindmapping is a presentation technique which uses symbols, colours, and words to create a map which may have a number of applications in the impact assessment process. The technique is useful at the structuring stage as for sorting out the perceptual process. It facilitates connecting facts and making synthesis of the actual knowledge. Usually, in the central part of the map is the headword from which thicker lines leading to keywords branch out. Then thinner lines, in turn, branch out from the thicker ones leading to words to which we ascribe, by association, less important information and so on. In this way the map can be built up infinitely. The second section has presented another application of this technique aimed at recreating intervention logic of the researched regulation, which is a starting point to structuring the frames of the whole impact research. Mindmapping could be used along with the other techniques described in this section. Decomposing the problem by branching on a map can be useful when the problem to be solved is being identified and when the problem solution is being developed. Using the map for summing up and sorting out the knowledge may be helpful at the stage of analyzing alternative solutions as well as solutions applied in other countries. As for ex post RIA mindmapping can be also significant while assessing informative obligations pursuant to a law. Obviously, it is possible to use mindmapping at other stages of impact assessment, which depends on decisions made by individuals engaged in this process.

Summing up What does MINDMAPPING involve? Using symbols, colours, words, arrows and texts, we create a map of associations, which makes it possible to sort out and sum up the knowledge. Thanks to using different colours and line thickness we can classify and differentiate the significance of particular elements, which could be regarded as general and specific, important and less important, etc. It is necessary to adopt a critical approach to relationships between particular elements of the map – to what extent they are casual and to what extent they depend also on other factors. Strengths

Weaknesses

• it is helpful in research designing and recreating intervention logic

• the map, especially if developed in short slogans can be, in fact, clear only to its author

• it allows to effectively associate facts • it facilitates knowledge synthesis • it provides a broad perspective for analyzed issues • it develops creative thinking Source materials on MINDMAPPING:  Buzan R., Buzan T. [2003], Mapy Twoich myśli, Lodz  Nast J. [2006], Idea Mapping: How to Access Your Hidden Brain Power, Learn Faster, Remember More, and Achieve Success in Business, New Jersey  http://ideamappingsuccess.com/

While mindmapping it is essential that we should: • use colors • use associations • use all the space on which the map is created • use different sizes of lines, pictures, letters • adopt a critical approach to causal links between particular map elements • use all available specialist computer software (both free of charge and chargeable) which facilitate mindmapping 3.3.3. Logical matrix A complementary tool to mindmapping is so-called logical matrix. It is mainly used for identifying or designing projects but it may be also relevant in view of impact assessment at the stage of analyzing public intervention logic. In this respect, the tool is used with reference to drafting a regulation to be

passed (ex ante) as well as the assessment of the objectives to be achieved by the legal act in force (ex post). The simplest matrix has a form of table with four columns and four lines. Particular lines of the matrix contain subsequent levels of the expected changes which have been obtained as a result of implementing a given intervention. The table columns refer subsequently to: areas of the expected changes; sources of information on whether the expected change has really occurred; assumptions which condition obtaining defined changes or effects (a specimen matrix is presented in Table 2.) Importantly, it is assumed that obtaining the changes at each level allows to obtain the changes at a higher level (Kierzkowski, 2002). Logical matrix can be used at the initial stages of the work on regulatory test and regulatory impact assessment when a problem to be solved is being identified as well as when the solution is being developed. Also, logical matrix can be used for conducting assessment of alternative solutions. With reference to ex post RIA, logical matrix seems to be a natural tool for analyzing the relevance of putting the law in force. Logical matrix can be one of the methods aimed at research structuring. Summing up

What does LOGICAL MATRIX involve? Its tabular form allows to analyze the logic of designed intervention as well as describe conditions in which the expected effects can be obtained. Thanks to it the feasibility analysis with reference to a given task is easier. Strengths

Weaknesses

• it allows to define initiatives to be undertaken in order to achieve the intervention objective

• its correct application requires practice

• it is helpful in checking whether a given solution will be feasible

• it may be too complicated and difficult to be analyzed

• it allows to reveal the cause-and-effect relationships at different intervention level Source materials on LOGICAL MATRIX:  Kierzkowski T. [2002], Ocena (ewaluacja) programów i projektów o charakterze społeczno-gospodarczym w kontekście przystąpienia Polski do Unii Europejskiej, Warsaw.  Pylak K. [red.] [2009], Podręcznik ewaluacji efektów projektów infrastrukturalnych, Warsaw, Ministry of Regional Development.

While creating a logical matrix it is essential that we should remember: • that it is a flexible tool – it can be adjusted to changing conditions • while putting objectives, to ascribe them to the proper level – general and specific ones • while putting indicators into the matrix, that they should be verifiable. 3.3.4. Analyzing existent data (desk research) Analyzing existent data is one of the basic methods used in research at the observation stage. It involves analyzing existent data from such sources as: 1) strategic and operational documents compiled by administration, 2) public statistics, 3) scientific and expert literature, 4) national, community and international legal regulations, 5) press articles. It allows to familiarize with a broader social and economic background for a given regulation. An unquestionable advantage of the method is easiness and fastness of obtaining data whose scope is often quite broad. Desk research is also a technique which does not require particularly large expenditures. However, the fastness of obtaining data does not correspond with the pace of processing the data obtained, which may turn out to be a time-consuming process. Additionally, it may happen that our activities are limited by lack of access to some data or by difficulty in obtaining them. The analysis of existent data is, in general, a starting point to almost every research process. The analysis of existent data can be particularly useful in the process of identifying a problem to be solved. In this case a data source can be the assessment of functioning a legal act (ex post RIA). This technique can be also applicable for developing a solution to a given problem as well as analyzing how the problem has been solved in other countries. Desk research could be also a supportive technique while identifying entities which are subject to a draft regulation and while analyzing changes in regulatory encumbrances resulting from the suggested solution after prior referring to the legal regulation in force. In terms of ex post RIA, analyzing existent data can complement the part describing a character of the draft impact on particular entities as well as the analysis of relevance of putting the law in force. In this respect using public statistics can be extremely useful. Secondary data can also serve for identifying costs and benefits of functioning a law in order to make a fur-

ther assessment as well as causes of differences between the assumed and the obtained results. Another application is for assessing informative obligations pursuant to the law and for analyzing opinions reported in relation to functioning the law. Summing up What does DESK RESEARCH involve? It involves analyzing existent data which derive from different sources. Strengths

Weaknesses

• easily and fast accessible data

• it can be time-consuming in view of analyzing accessible information and data

• it does not generate high costs

• it is possible that there is a limited access to some data or that the data do not exist

• it is possible to obtain a wide range of data Source materials on DESK RESEARCH:  Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warsaw.  Silverman D. [2012], Prowadzenie badań jakościowych, Wydawnictwo Naukowe PWN, Warsaw.  Silverman D. [2012], Interpretacja danych jakościowych, Wydawnictwo Naukowe PWN, Warsaw.

While analyzing data it is essential that we should: • use national and international public databases • continuously verify timeliness of the obtained data • have a critical approach to reliability of the obtained data 3.3.5. Individual In-Depth Interview (IDI) IDI is an interview with a responder conducted by a researcher according to a script or in a form of the interviewee’s free speech. It belongs to a group of key qualitative research methods which allow to familiarize with the interviewees’ opinions on a given subject (as a rule individuals directly engaged in a given intervention). The method is useful at the stage of observation and it provides in-depth information on how a given intervention has been implemented in reality. Among unquestionable advantages of IDI there is relatively a short time

in which it is conducted and a possibility of analyzing a given issue in detail. These advantages are possible thanks to providing an interviewee with a chance of free expressing their viewpoints. However, it is necessary to remember that the interview can be used only to examine a limited number of responders. In some cases it is possible to conduct an in-depth interview with two or three individuals at a time (respectively dyad or triad) or in the form of a telephone conversation. Individual interviews are the second most popular (after analysis of existent data) research technique. It allows to better understand the logic and effects of a given regulation, thanks to obtaining an in-depth knowledge directly from individuals engaged in a given process. The IDI formula, thanks to obtaining detailed statements from experts/ stakeholders in a given area, allows to use the technique as a complement to some RIA parts. It will definitely be relevant at the stage of collecting information on the character of a problem while it is being identified. The interview can be applicable when the ways of conducting tasks challenged by a regulation are being analyzed as well as others countries’ experiences are confronted. With reference to ex post RIA, we can use the in-depth interview for assessing the relevance of enforcing a given regulation as well as for analyzing the feasibility of reducing informative obligations. IDI is also successfully applicable at the stage of collecting opinions reported in relation to functioning a law. Summing up What does INDIVIDUAL IN-DEPTH INTERVIEW involve? IDI is conducted by a moderator who, thanks to properly prepared questions, obtains detailed information on a given phenomenon from an individual who possess a knowledge of this subject. We can also decide on conducting an in-depth interview with 2 individuals at a time or on a telephone interview. The interview can complement data used in the IA process obtained by other techniques. Strengths • a short time of conducting the research

Weaknesses • this method is costly and time-consuming in view of processing data

• a responder can express their opinions in their own words • in this way only a small number of responders can be examined effectively • a possibility of discussing complicated issues in detail

Source materials on INDIVIDUAL IN-DEPTH INTERVIEW:  Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warsaw.  Silverman D. [2012], Prowadzenie badań jakościowych, Wydawnictwo Naukowe PWN, Warsaw.  Silverman D. [2012], Interpretacja danych jakościowych, Wydawnictwo Naukowe PWN, Warsaw

While conducting IDI it is essential that we should: • prepare an interview script with the main questions to a responder. It is advisable to complement each main question by detailed questions aimed at clarifying the responder’s opinion on the major issues (if they were not touched by the responders themselves) • listen to the interviewee attentively and with interest • allow the interviewee finish their speech – do not interrupt it or do not try to finish answering for them • ask for details, e.g. why something has occurred, how it has happened and what results from it • ask the interviewee for their consent for audio recording the meeting in order to make a minute of the interview or its transcription • make notes regardless of consent for recording • inform the interviewee about the research anonymity 3.3.6. Focus Group Interview (FGI) FGI is an interview of a properly selected group of responders which is conducted by a moderator according to a script. It is one of qualitative research methods used at the stage of observation which allows to take an in-depth look at a given issue and get to know responders’ subjective opinions. FGI will be relevant in the case of the problem on which there is no unambiguous opinion. Thanks to using this method it is possible to confront different opinions on a given subject. FGI will be also applicable if we have a larger group of responders who can contribute to obtaining a respectively large amount of information in a relatively short time. In view of this, we will use FGI at the stage of identifying a problem; while the ways of conducting tasks challenged by a regulation are being analyzed as well as while analyzing other countries’ experiences. FGI can also contribute to obtaining data within ex post RIA, especially as for assessing the relevance of implementing a given regulation. The responders’ knowledge and experiences can be also referred to within the feasi-

bility of reducing informative obligations. Also, FGI can be applicable as for collecting opinions reported in relation to functioning a law. Summing up What does FOCUS GROUP INTERVIEW involve? It is a moderated interview of a properly selected group of responders aimed at an in-depth discussion on a given issue. Strengths

Weaknesses

• relatively low costs

• problematic data analysis

• we obtain a large amount of information in a short time

• possible difficulties in selecting and then gathering responders

• a chance of familiarizing with responders’ subjective opinions and viewpoints on the problems on which there are no unambiguous opinion

• necessary participation of an experienced moderator

• value added related to interaction within the group Source materials on FOCUS GROUP INTERVIEW:  Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warsaw.  Silverman D. [2012], Prowadzenie badań jakościowych, Wydawnictwo Naukowe PWN, Warsaw.  Silverman D. [2012], Interpretacja danych jakościowych, Wydawnictwo Naukowe PWN, Warsaw.

While conducting FGI it is essential that we should: • prepare a list of questions we want to ask • ‘break the ice’- make an introduction to the subject, ask the interviewee to introduce themselves or prepare tickets with their names on, etc. • control the course of the conversation – control timing, take care of not dominating the discussion by a strong individual • control our reactions- avoid value judgements, keep eye contact, nod from time to time, encourage to expand a discussed plot if we see there is a chance for it • possibly think about hiring an experienced moderator • ask the interviewees for their consent for audio recording the meeting in order to make a minute of the interview or its transcription • inform the participants about research anonymity

83 3.3.7. Case study Another method suggested to be used at the stage of observation which facilitates revealing in-depth information on a given intervention in practice is case study. Thanks to analyzing a specific case at the ex post RIA we obtain indepth information on impact mechanism and unexpected effects of functioning a regulation. It allows for better understanding the processes in progress and thanks to using data from a few independent sources (e.g. interviews, document analyses) it contributes to an in-depth analysis of the researched phenomenon. Case study is applicable for analyzing regulatory impact if a pilot scheme for functioning a given legal act has been conducted. As for ex post RIA, case study can be useful with reference to analyzing the relevance of implementing a law in order to make a comparative summary concerning the situation after and before its implementation. Case study is also a supportive method for identifying a cause of differences between the assumed and obtained results (when e.g. in one area an intervention has brought the results different from the results in other areas). Summing up

What does CASE STUDY involve? It is a moderated interview of a properly selected group of responders aimed at an in-depth discussion on a given issue. Strengths

Weaknesses

• it is possible to obtain a great deal of information on, among others, objectives, motivation and background in which an intervention is conducted

• it is difficult to generalize the results in view of society

• it allows to ‘visualize’ conclusions from the research on the basis of examples

• risk of the researcher’s subjectivity

• it serves for getting to know mechanisms which generate changes

• the researcher’s great commitment is necessary

Source materials on CASE STUDY:  Flyvbjerg B. [2006], Five Misunderstandings about Case Study Research, Sage Publications. Gaffey V. [2009], Case studies in the framework of ex post evaluation, 2000–2006: expectations and experiences  of the DG for Regional Policy, Brussels.  Stake, R. E. [1995], The Art of Case Study Research, Thousand Oaks: Sage.

While preparing case study it is essential that we should: • not rely only on existent data analyses – let us use other techniques, e.g. individual interviews • illustrate the study with photographic documentation when specific projects are presented

3.3.8. Mystery client/mystery shopper This method is usually associated with research on the quality of client service involving visits to researched outlets (e.g. shops, banks, etc.), observation and assessment made by a client on the basis of a compiled questionnaire. The observation is finished at the moment of developing a report which contains the ‘client’s remarks. However, this methods turns out to be also useful as for impact assessment, e.g. with reference to an intervention which involves providing citizens with public services both in reality and via the Internet. In this case the client is replaced by the researcher. Thanks to the mystery client method we can examine not only the actual level of the change in the way of servicing the citizen due to enforcing a regulation but also test the utility of IT solutions aimed at ensuring citizens access to these services. This method is mainly used for the research on the quality and effectiveness of the implemented regulation (in fact, the service quality and effectiveness). The mystery client method will be mainly applicable at the stage of identifying a problem to be solved and it will be regarded as a source of knowledge of the problem (a form of verifying whether and to what extent a given problem exists). As for ex post RIA, ‘the mystery client’ can be, in turn, used with reference to analyzing the relevance of implementing a law as well as at the stage of identifying a cause of differences between the assumed and obtained results.

Summing up What does MYSTERY CLIENT involve? This research involves visiting by a researcher equipped with a specially designed questionnaire outlets which are subject to the research and making the observation and assessment of service. Strengths

Weaknesses

• it does not focus on statements and opinions but is rather • it is limited by a set of regulations which must be obeyed reporting on the progress of an authentic situation while it is being conducted • it makes it possible to test the introduced solutions in practice Source materials on MYSTERY CLIENT:  CIiPKZ w Tarnowie, Informacja o zawodzie. Tajemniczy klient. http://wup-krakow.pl/uslugi-rynku-pracy/poradnictwo-zawodowe/informacje-zawodowe-1/o-zawodach-pliki/tajemniczy klient.pdf [dostęp: 17.04.2015]  Maison D., Noga-Bogomilski A. (red.), (2007), Badania marketingowe. Od teorii do praktyki, GWP, Gdansk,. Maison D., (2010), Jakościowe metody badań marketingowych. Jak zrozumieć konsumenta, Wydawnictwo Naukowe PWN,  Warsaw 2010.

While using the mystery client methods it is essential that we should: • rather rely on the experience of companies specializing in it • take account of the fact that the scope of the information we want to obtain ought to be in line with the guidelines defined in the code of the European Society for Opinion and Marketing Research (ESOMAR) 3.3.9. Cost-benefit analysis (CBA) This method, useful at the analysis stage, is aimed at assessing whether the costs of a given intervention will be commensurate with the benefits brought by it. As a rule the method is based on market prices, however when their defining is difficult or even impossible (e.g. as for social or environmental costs) estimated prices are taken into consideration. The conclusions following CBA can contribute to maximizing benefits and minimizing costs as well as declaring a given intervention uneconomical if the costs are too high. As a rule, CBA consists of 7 stages (EC, 2014): 1) background description, 2) objective definition, 3) project identification, 4) technical feasibility and environmental sustainability, 5) financial analysis, 6) economic analysis, and 7) risk assessment.

The CBA method is one of the main tools used in ex ante RIA. In practice, the most important problem related to its application is lack of sufficient data necessary for a reliable estimation of the real social and economic costs. However, this method can be a tool illustrating the results of comprehensive impact research. As for impact assessment, this method is applicable with reference to cost-benefit analysis of solution alternatives which have been taken into consideration and of the entities being subject to the project. CBA is the most helpful for analyzing regulatory impact. With reference to ex post RIA, CBA can be used for indicating the costs and benefits related to functioning a law and for assessing informative obligations pursuant to a law. Summing up

What does COST-BENEFIT ANALYSIS (CBA) involve? Strengths

Weaknesses

• it makes it easier to sum up overall costs and benefits of ex ante regulatory project

• in order to be conducted reliably, it needs a number of research works to have been done earlier

• it allows to express all costs and benefits in monetary terms.

• it may not take account of all aspects of designed regulation

Source materials on CBA:  DG REGIO, European Commission, Evalsed Sourcebook: Method and techniques. http://ec.europa.eu/regional_policy/sources/docgener/evaluation/guide/evaluation_sourcebook.pdf  DG REGIO, European Commission [2014], Guide to cost-benefit analysis of investment projects. http://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/cba_guide.pdf

While analyzing costs and benefits it is essential that we should: • not neglect the importance of background analysis – further reliability of forecasts concerning the stakeholders of costs and benefits may depend on the accuracy of assessing macroeconomic and social conditions • take care of the fact that economic results considered in the analysis should take account of specific regulatory objectives • juxtapose the option analysis and a counterfactual scenario • remember that CBA is a source of information which can be verified while real intervention effects are being researched

87 3.3.10. Expert panel Expert panel is a meeting of a group of experts who following a discussion led by an experienced moderator work out a joint opinion on a given subject. This technique can be successfully used both for analyzing data, when it is necessary to obtain specialist expertise on more complex problems, and for drawing conclusions. However, the technique needs a qualified moderator who will ensure the proper progress of discussion and prevent strong individuals from dominating the group. The drawback of expert panel could be subjectivity of the participants’ opinion. Despite the above objections this technique can be useful at the stage of analyzing a problem. It is also worth engaging experts in discussion on solving a given problem as well as on analyzing how the problem has been solved in other countries. As for ex post RIA, expert panel can be helpful in analyzing the relevance of implementing a law. Summing up What does EXPERT PANEL involve? The technique involves meeting a group of experts in one field held with the aim of discussing issues within one area. The discussion is moderated by a qualified person. Strengths

Weaknesses

• it is helpful for analyzing very complex problems which need specialist expertise

• it is time-consuming

• it allows to verify conclusions drawn in the course of discussion

• it needs a moderator who will lead discussion skillfully • discussion may be dominated by individuals • risk of subjective opinions expressed by participants

Source materials on EXPERT PANEL:  DG REGIO, European Commission, Evalsed Sourcebook: Method and techniques. http://ec.europa.eu/regional_policy/sources/docgener/evaluation/guide/evaluation_sourcebook.pdf Pylak K. [red.] [2009], Podręcznik ewaluacji efektów projektów infrastrukturalnych,  Warsaw, Ministry of Regional Development.

While organizing expert panel it is essential that we should: • prepare a list of questions to be asked • control the course of discussion - watch out for timing, prevent a strong individual from dominating discussion • possibly think about hiring an experienced moderator • ask participants for their consent for audio recording the meeting in order to make a minute of discussion or its transcription • inform participants about research anonymity. 3.3.11. Benchmarking This method involves impact assessment of a given intervention by juxtaposition of undertaken initiatives and analogous examples (national and foreign) which have become successful. As for impact assessment, it will allow to define strengths and weaknesses of a drafted regulation and identify new alternative solutions. It will make it possible to use good practices, including those from other countries, after adopting them to the needs of a given regulation. The technique can be successfully applicable within impact assessment provided that it is possible to skillfully identify the examples which are referred to. Another hindrance may be the problem of ‘transplanting’ effective solutions into national ground. In view of its character, benchmarking is applicable for analyzing how a problem has been solved in other countries. Contacting officials or institution representatives from other countries can be an effective way of obtaining information on possible effects of a drafted regulation. Therefore it is worth referring to foreign experiences in the cases of analyzing he change in regulatory incumbrances resulting from the proposed solution and regulatory impact analysis. Benchmarking also proves to be useful for ex post RIA, as it can help identify causes of differences between the assumed and obtained results.

Summing up What does BENCHMARKING involve? While conducting regulatory impact assessment, we refer to similar experiences, including those from other countries, in view of specificity of a designed intervention in order to identify possible effects of a drafted regulation. Strengths

Weaknesses

• it allows to identify solutions which have become success- • possible difficulty in identifying the examples which ful or failed in similar conditions could be reference points • it helps reveal a broad background of the processes which refer to a given regulation

• possible allegation of faithful ‘copying’ foreign solutions • possible difficulty in adjusting interesting solutions to national specificity.

Source materials on BENCHMARKING:  Bogan C. E., English M. J. [2006], Benchmarking jako klucz do najlepszych praktyk, Onepress, Gliwice. Ziębicki B. [2000], Informacyjne wspomaganie benchmarkingu, Zeszyty Naukowe Akademii Ekonomicznej w Krakowie,  No 534, Cracov.

While conducting benchmarking it is essential that we should: • treat the method as a way of learning and a source of inspiration rather than a method providing ready-made solutions to be directly transferred to our ground • remember that the identified examples can be shown as case studies

3.4. Characteristics of selected quantitative research methods and techniques A quantitative approach used in social research, including evaluation, differs basically from the qualitative approaches which have been presented above. The latter provide most frequently a set of information which is rather unlikely to be translated into figures18 . As for quantitative approaches, their real essence is, in fact, a possibility of quantifying phenomena. A particularly distinctive characteristic of quantitative approaches is that they aim at identical results 18

I t is worth paying attention to more and more popular trend of developing tools for quantitative analysis of qualitative data.

achieved by every researcher under the same research conditions and circumstances (Lisowski et al., 2008). This desirable property of quantitative approaches is referred to as research reliability. On the whole, the purpose of research based on quantitative approaches is "to formulate statistical theorems, i.e. statements describing properties of statistical distribution of one or more variables in a researched group or in several comparable populations" (Lisowski et al., 2008). In other words, the subject of quantitative research is a detailed analysis of the characteristics of selected communities which are in the researcherâ&#x20AC;&#x2122;s interest. Quantitative approaches along with statistical tools used by them are, in fact, a certain form of reducing information â&#x20AC;&#x201C; a wide range of data is replaced in the analysis process by a few figures (percentage, average figures, correlation coefficients, etc.) Depending on whether a researcher possesses full or only partial (a sample) information on a given population, slightly different procedures of analyzing the accessible data will be used. In the former case the analysis will have a character of statistical description, whereas in the latter it will be necessary to use the tools of the so-called inductive statistics which involves grossing-up procedures aimed at inferences based on sample research with reference to a population as a whole (Blalock, 1977). The inferring procedure is based on mathematical probability theory, which means that within conducted analyses, selected variables (indicators) are quantified and then assessed in view of the probability of their mirroring real parameters in a population. However, it is necessary to remember that in the case of statistical inference the way of selecting a sample is especially of great importance. In this respect the key role is played by random sampling. In practice, an individual conducting IA very often faces lack of information on specific features of the population which is subject to a given regulation. Also, analysts developing IA are limited by the scope of information collected within public statistics or an internal system of monitoring and reporting in public units. At the same time it is known that a given regulation can have a real impact on various dimensions of functioning units which will be (or have already been) subject to it. In order to find out a quantified value of a given feature or its statistical distribution in a population it is necessary to obtain the desirable information following additional empirical research.

Quantitative research data can be useful especially if we are interested in th underlying value of a certain feature in population – i.e. prior to passing a legal act. After the regulation is put in force the quantified value of selected indicators can be subsequently monitored and then used at the stage of ex post assessment. Moreover, quantitative analyses can be an important tool in the process of verifying different hypotheses e.g. with regard to differentiation of appearing a given characteristic or a set of characteristics in groups which are to be subject to the regulation in progress. At the stage of analyzing data various techniques of verifying differentiation of characteristics and their collocations can prove to be helpful. Also, different prognostic tools and segmentation techniques will be invaluable. The described techniques of quantitative research can be successfully applicable for collecting structured opinions in the social consultation process. Similarly to qualitative research, the text below presents selected techniques of collecting data for the research based on quantitative approaches. In the most general sense these approaches can be divided into two types. One uses data collected in a form of questionnaire-based interview (CATI, CAPI, PAPI) whereas the other one involves responders providing data and information for researchers without third parties such as, for example, interviewers (CAWI, auditorium survey). The absence of an interviewer has both disadvantages as well as advantages. On the one hand, the interviewer watches out for a responder that they should get acquainted with all questions according to the intention of the researcher preparing the research. The interviewer also ensures that the right answer indicated by the responder should be ticked in the questionnaire. On the other hand, the presence of an interviewer during the survey may result in the socalled ‘interviewer effect’, which refers to the situation in which the responders’ answers are under direct or indirect influence of the interviewers (e.g. interviewers can suggest desirable answers or impose their ways of understanding questions on responders, etc.) Each presented technique of collecting data is related to a different procedure. Some require using additional technological tools, including IT. Their practical applications will be conditioned by the accessibility of the researched units (so-called sampling frame) and the researcher’s access to specific contact

details in particular. The discussed techniques generate also other financial costs, which may turn out to be an essential factor while making a selection.

3.4.1. CATI technique

CATI stands for Computer-Assisted Telephone Interview and is a technique for conducting surveys which involve interviewing on the phone with the use of the computer and specialist software supporting particular stages of conducting the research. The interviews are conducted in specially-designed research workplaces (so-called CATI studios) As for CATI, an interviewer contacts an individual selected by an IT system on the phone, arranges an interview and, if successful, conducts uploading the responder’s answers to the database. If an interview is unlikely to be conducted, the reason for this is inserted into the system. Among the most important advantages of CATI are standardization of research process, fastness of conducting the research and relatively low costs (as compared to CAPI or PAPI). Moreover, the IT system used in CATI provides: • a possibility of eavesdropping and watching each interviewer’ work simultaneously by research supervisors (at any time of interviewing), which, as a result, contributes to the higher quality of conducting the research – possible doubts or problems are immediately pocked up and solved • a fully-automated, independent of interviewers, way of selecting responders • a possibility of arranging interviews in line with responders’ preferences (arranging an interview at any time) • a standardized system of classifying causes of unsuccessfully conducted interviews • a possibility of breaking an interview at any time and its continuing at any time indicated by a responder • an entry of the research background data such as the time and overall length of conducting the interview and details concerning the interviewer.

Summing up What does CATI involve? The technique involves collecting data and information with the use of IT tools. Strengths

Weaknesses

• standardization of the research process

• necessity of possessing telephone contact details

• possibility of conducting research on different subjects

• possibility of conducting research within well-established attitudes (touching more complicated issues may result in research artifacts)

•short time of conducting research

• difficulty in keeping responders’ constant attention and concentration.

• relatively low research costs • high flexibility of conducting an interview (possibility of arranging an interview at any time, and/or breaking it at any moment) • easiness of managing interviewers’ work. Source materials on CATI:  Babbie E. [2008], Badania społeczne w praktyce, Wydawnictwo Naukowe PWN, Warsaw.  Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warsaw.  Frankfort-Nachmias Ch., Nachmias D [2001]. Metody badawcze w naukach społecznych, Zysk i S-ka, Poznań.  Haber A., Szałaj M. [red.] [2007], Ewaluacja ex post. Teoria i praktyka badawcza, PARP, Warsaw.  Olejniczak K., Kozak M., Ledzion B. [red.] [2008], Teoria i praktyka ewaluacji interwencji publicznych. Podręcznik akademicki, Wydawnictwa Akademickie i Profesjonalne, Warsaw.  Sztabiński, P.B., Sawiński, Z i Sztabiński, F. [2005], Fieldwork jest sztuką, Wydawnictwo IFiS PAN, Warsaw.

3.4.2. CAPI technique CAPI stands for Computer-Assisted Personal Interview and is a form of survey which uses, similarly to CATI, the computer and specialist software supporting the data collecting process. Contrary to CATI, CAPI is based on an interview conducted by an interviewer in person , which means face to face with a responder. The latter’s answers are noted down continuously when the interview is in progress and the-

refore conducting the research in this way takes usually a longer time as compared to techniques which do not need direct contact with the responder and is relatively more expensive. The greatest advantages of the CAPI technique involve standardization of the data collecting process and greater control of real research participants. Also, direct contact with a responder makes it easier to present additional materials and information, which would not be very likely to be passed on ‘remotely’. Furthermore, using an IT system for conducting the research facilitates: • conducting personal interviews with the computer use in the place indicated for the research in the base or in another one after prior arranging it with a responder • conducting interviews complemented by sound and visual prompts e.g. pictures, films, etc. • relatively prompt access to research results • automated and independent of the interviewer selection of research responders • unambiguous identification of each interviewer through the interviewer authentication system • indirect control of interviewers’ work by registering selected occurrences during the interview (e.g. the time and length of conducting each interview, time of exposing supportive materials, laps) • interrupting the interview at any time, arranging it or restarting/continuing it at any time • supervision and control over conducting the research. The IT system for CAPI should make it easier to report immediately on the progress of particular interviewers’ work or the sampling level , etc. A certain variety of CAPI is the CASI (Computer-Assisted Self Interviewing) technique. It involves inserting data by a responder into the shared computer on their own without the interviewer’s presence.

Summing up What does CAPI involve? CAPI is a technique of conducting a personal interview when an interviewer uses the computer (or alike device) for taking down the responder’s answers. Strengths

Weaknesses

• standardization of the research process

• relatively a long time of collecting data (as compared to other quantitative research techniques- CATI, CAWI)

• possibility of conducting research on different subjects

• higher costs

• high flexibility of conducting an interview (possibility of arranging an interview at any time, and/or breaking it at any moment)

• relatively a smaller sense of responders’ anonymity

• easiness of managing interviewers’ work. Source materials on CAPI:  Babbie E. [2008], Badania społeczne w praktyce, Wydawnictwo Naukowe PWN, Warsaw. Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warsaw.   Frankfort-Nachmias Ch., Nachmias D [2001], Metody badawcze w naukach społecznych, Zysk i S-ka, Poznań. Haber A., Szałaj M. [red.] [2007], Ewaluacja ex post. Teoria i praktyka badawcza, PARP, Warsaw.  Olejniczak K., Kozak M., Ledzion B. [red.] [2008], Teoria i praktyka ewaluacji interwencji publicznych.  Podręcznik akademicki, Wydawnictwa Akademickie i Profesjonalne, Warsaw. Sztabiński, P.B., Sawiński, Z i Sztabiński, F. [2005], Fieldwork jest sztuką, Wydawnictwo IFiS PAN, Warsaw. 

3.4.3. PAPI technique PAPI – Paper and Pen Personal Interview is the most traditional form of conducting quantitative field research. An interviewer conducts an interview with a responder using a questionnaire printed before. Despite growing popularity of data-collecting research methods based on computer technology, PAPI still proves to be useful and successful in some selected situations (e.g. when it is necessary to conduct a large number of interviews at a limited amount of time) One unquestionable advantage of this technique is that it does not need any defined background necessary for starting the research. On the other hand, the information obtained within PAPI must be, after finishing the interview, transformed to an electronic database for their further statistical analysis. This significantly affects the length of the research process.

While designing a research tool for PAPI, it is necessary to pay relatively more attention to the questionnaire clarity, especially when it assumes that optional questions are asked depending on the responder’s specific characteristics. In techniques based on specialist software the questions are controlled by an IT system and they are asked to responders according to some ‘paths’ of going through the questionnaire which have been defined (programmed) before (so-called filters). Summing up

What does PAPI involve? PAPI involves conducting personal interviews using a paper questionnaire. Strengths

Weaknesses

• low technological requirements for starting research

• greater difficulty in controlling interviewers’ work

• relatively low costs as compared, e.g. to CAPI

• time-consuming procedure of data encoding • no chance of presenting additional visual or sound elements • relatively long time necessary for research preparation (especially a questionnaire)

Materiały źródłowe nt. techniki PAPI  Babbie E. [2008], Badania społeczne w praktyce, Wydawnictwo Naukowe PWN, Warsaw. Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warsaw.   Frankfort-Nachmias Ch., Nachmias D [2001], Metody badawcze w naukach społecznych, Zysk i S-ka, Poznań. Haber A., Szałaj M. [red.] [2007], Ewaluacja ex post. Teoria i praktyka badawcza. PARP, Warsaw.  Olejniczak K., Kozak M., Ledzion B. [red.] [2008], Teoria i praktyka ewaluacji interwencji publicznych. Podręcznik akademicki.  Wydawnictwa Akademickie i Profesjonalne, Warsaw. Sztabiński, P.B., Sawiński, Z i Sztabiński, F. [2005], Fieldwork jest sztuką. Wydawnictwo IFiS PAN, Warsaw 

3.4.4. CAWI technique CAWI stands for Computer-Assisted Web Interview and serves for surveys in which an interview is conducted through a website. The questionnaire is filled by an individual who has access to the proper questionnaire link (both uni-

versal and individual links may be accessible). The questionnaire can be also password-protected. As for CAWI, responders are invited electronically. If individual links are used, the IT system continuously records responders’ answers. It also makes it possible for a responder to record their answers on their own, break the interview at any time and return to it when conveniently. The greatest advantages of CAWI include, similarly to CATI and CAPI, the research process standardization (the logic of filling the questionnaire is defined and controlled by the IT system in use), promptness of conducting the research and low operational costs regardless of the size of defined research sample, which is really important. CAWI is also convenient to responders as they can take part in the survey at the most comfortable moment for them. This technique, by definition, is not exposed to risks of interviewers’ errors. Moreover, the IT system used in CAWI provides: • continuous monitoring the research being conducted, including the progress of filling particular questionnaires • designing questions referring to a database – responders can be presented with personalized content (e.g. of target indicators) • safe and multiple attempts of logging to the website containing the survey questionnaires • entry of data concerning the research background in the base (the time and length of conducting the survey). Summing up

What does CAWI involve? A responder fills in an Internet questionnaire in person. Strengths

Weaknesses

•short time necessary for preparing the survey

• difficulty in obtaining a satisfactory level of the response rate

• short time of conducting the survey

• lack of computer literacy in some population groups

• no interviewer effects

• difficulty in ensuring and controlling the sample representativeness

• relatively low operational costs (regardless of the size of researched population)

• limited control of those who fills in the questionnaire

98 • chance of using additional multimedia elements

• necessity of possessing e-mail data base

• great flexibility – a responder decides when they fill in the questionnaire Source materials on CAWI:  Babbie E. [2008], Badania społeczne w praktyce, Wydawnictwo Naukowe PWN, Warsaw. Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warsaw.   Batorski, D., Olcoń, M. [2006], Prowadzenie badań przez internet: Podstawowe zagadnienia metodologiczne. Studia Socjologiczne, 3 (182), Warsaw. Haber A., Szałaj M. [red.] [2007], Ewaluacja ex post. Teoria i praktyka badawcza, PARP, Warsaw.  Haber A., Szałaj M. [red.] [2009], Ewaluacja wobec wyzwań stojących przed sektorem finansów publicznych, PARP, Warsaw.  O lejniczak K., Kozak M., Ledzion B. [red.] [2008], Teoria i praktyka ewaluacji interwencji publicznych. Podręcznik akademicki,  Wydawnictwa Akademickie i Profesjonalne, Warsaw. Sztabiński, P.B., Sawiński, Z i Sztabiński, F. [2005], Fieldwork jest sztuką, Wydawnictwo IFiS PAN,Warsaw. 

3.4.5. Auditorium survey Auditorium survey is a data-collecting technique which involves handing in a paper questionnaire to a group of individuals gathered in one place who are to fill it on their own. This place can be mostly a classroom, a workspace or an auditorium, etc. Having been filled, the questionnaires are collected by a designated person and then encoded in a selected data-collecting IT system. In view of the specificity of conducting auditorium survey, it is a good tool to be used during independently organized meetings or events because in this way all organizational problems related to gathering a defined group of individuals in one place are avoided. An important distinctive feature of this technique as compared to those presented above is high standardization of conditions in which the interview is conducted because, as a rule, all responders take part in the survey at the same time and in the same place.

Summing up What does AUDITORIUM SURVEY involve? Auditorium survey involves using a paper questionnaire to be filled in by responders gathered in one location selected by the researcher. Strengths

Weaknesses

• low operational cost

•paper questionnaire answers must be encoded

• short time of collecting data

• responders must be gathered in one location (logistic difficulty)

• very high response rate (most frequently up to 100%)

• long time of encoding the questionnaire results

• no interviewer effects

• long time of encoding the questionnaire results

• relatively high sense of responders’ anonymity

• limited control of those who fills in the questionnaire

• high level of standardization of research condition

• necessity of possessing e-mail data base

Source materials on AUDITORIUM SURVEY:  Babbie E. [2008], Badania społeczne w praktyce, Wydawnictwo Naukowe PWN, Warsaw. Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warsaw.   Frankfort-Nachmias Ch., Nachmias D [2001]. Metody badawcze w naukach społecznych. Zysk i S-ka, Poznań. Sztabiński, P.B., Sawiński, Z i Sztabiński, F. [2005], Fieldwork jest sztuką. Wydawnictwo IFiS PAN, Warsaw. 

3.5. Using public consultation results in the process of impact assessment Consultations are mostly aimed at improving the quality of the regulation being drafted. The exchange of views on our work with representatives of different social groups contributes also to building up and consolidating social capital. Thanks to including stakeholders in the consultation process as early as possible we can obtain invaluable substantive guidelines. Their scope depends on how wide a range of entities will be included in the consultation process. In this way we get to know their points of view, which may be helpful while drafting a law and adjusting it better to the real needs of the entities which

100

will be subjects to the regulation. Consultations also prevent us from errors we could have made if they had not been organized and make their participants identify better with our initiatives both at the beginning of the process and later when it is in progress. As for organizing consultations, it is necessary to take account of the guiding principle: We consult because we WANT not because we HAVE TO

The quality of information obtained through consultations is extremely important in view of the impact assessment which will be conducted later. Therefore it is necessary to make all possible efforts to obtain the data which could be used in the subsequent processes. Consultation results complement significantly data obtained for impact assessment through the methods and techniques described in section 3. However, it is worth remembering that by using the same methods and techniques it is possible to conduct consultations even in a more efficient way than a traditional writing formula. In this case a selection of proper tools is the sine qua non condition of providing sound evidence for our debate. We can use either qualitative or quantitative methods. Focus Group Interview, brainstorming or expert panel (if we have identified a small, homogenous group of partners) can be particularly useful. Organizing consultation meetings traditional moderation can be complemented by mindmapping. An invaluable source of information for the IA process as well as for consultations is quantitative research. It should be stated that in view of ensuring the reliability of obtained data, using research methods is more meaningful. Gaining information through consultations conducted in a traditional way, which is collecting the stakeholdersâ&#x20AC;&#x2122; opinions, poses a risk of obtaining standpoints difficult to be verified, then codified and finally used in the analytical process. In view of conducting IA it is necessary to ensure close cooperation of individuals in charge of social consultations and individuals conducting impact assessment. Information obtained through consultations can become a significant source of knowledge for impact assessment in progress. Examples of properly

101

designed surveys or qualitative research (personal interviews/focus group interviews) described in this section can account for the most important pieces of information for cost-benefit assessment. In this respect, however, it is necessary to ensure cooperation as early as the stage of designing a tool itself.

3.6. Towards causal model

One of research designs aimed at conducting impact assessment, especially at the ex post RIA stage includes experimental and quasi-experimental methods. They have been increasingly used for evaluation research within Cohesion Policy. With reference to evaluation, they are included in research trend based on the so-called counterfactual scheme (counterfactual impact evaluation). A wide range of Polish experiences in using quasi-experimental methods points at their possible application for regulatory impact assessment. This kind of research is used to find a reliable answer to the question: Has conducting our intervention had a real impact on a defined dimension of social and economic situation? Counterfactual assessment allows to define whether there is the cause-and-effect relationship between our initiatives and observed social and economic changes. It also makes it possible to define quite precisely what would happen if we weren’t conducting out intervention (if we weren’t putting a regulation in force). At this point it is worth referring to Part two of this handbook to remind that each public intervention – a programme, a project or a regulation in particularcould be perceived as a kind of theory of change. While making a decision about its implementation it is widely assumed that defined initiatives undertaken in defined conditions with the use strictly defined resources will contribute to solving certain defined problems. In view of this, the role of evaluation research as well as IA analyses is to check whether in the light of the obtained effects a given programme theory should be either sustained or rejected (Górniak, 2007). Evaluation and IA (especially at the ex post RIA stage) can be perceived as a process of verifying theories of the assessed programmes. Experimental and quasi-experimental methods are aimed at assessing direct effects brought by implementing a specific regulatory intervention as compared to a sheer hypothetical situation in which the regulation is put in force.

102

The core of the research is to define whether the changes which have been made in the addressees of a legal act (individuals, entities, local government units which are subject to the legislative act) such as employing the unemployed, an increase in companiesâ&#x20AC;&#x2122; turnovers, etc. result from: 1) putting regulation in force 2) other outer occurrences (economic situation, an overall increase in employment in the economy, etc.) The research design should compare the observed changes in the situation of the regulation addressees with an analogous situation in which the regulation was not enforced. And at this point one objective problem arises â&#x20AC;&#x201C; we have difficulty observing impact in parallel realities (Bienias, 2010, presentation at Warsaw School of Economics). In few cases we can define this impact by simply analyzing the addresseesâ&#x20AC;&#x2122; situation before and after implementing the regulation. The obtained difference can be then interpreted as the impact of our regulation. However, such an inference is possible only if we are absolutely sure that no change has been made at all but for the regulation. In other words, it should have been assumed that there is no natural dynamics in the social and political reality. This is a very strong assumption, which is hardly ever fulfilled (Bienias, 2010, presentation at Warsaw School of Economics). Take a look at the diagram below presenting the impact of the regulatory intervention which is going to be assessed by us. Diagram 5. Intervention impact (1)

Intervention impact B

Observation

1-0 implementation of regulation before implementation

Source: by authors

1-1

time (x)

after implementation of regulation

Employment level in LGU

Average annual enterprise revenue

Observation

1-0 implementation of regulation before implementation

1-1

time (x)

after implementation of regulation

103

While analyzing the chart, letâ&#x20AC;&#x2122;s take a specific regulatory intervention regarding special economic zones as a simplified example. In this case a counterfactual scheme can be used at the two levels : a) Specific enterprises which are the support addressees (companies located in SSE) b) Local government units (municipalities/districts where SSE are located) In case a) we assess whether the fact of obtaining tax exemption in the economic zone has influenced the situation of the enterprises located in it. We can decide on quantifying competitiveness by the indicator which is the average revenue of the company in the calendar year. The moment before the zone came into existence, the companies gained the average annual revenue at the level of Y0 (point A, Chart 2). Some time after the regulation has been in force (t=1) we check what revenues have been gained by the enterprises located in special economic zones. Following the regulation implementation, the average annual revenue of the supported companies has increased to Y1 (point B Chart 2). Case b) presents an analogous situation (chart 2). We assess here whether the fact of establishing a special economic zone in a municipality has influenced the social and economic situation within a given local government unit. We can decide to quantify the social and economic situation by the indicator defining the employment level. The moment before the zone came into existence in the municipality, the employment level amounted to Y0 (point A, Chart 2). Some time after the regulation has been in force (t=1) we check the employment level in the local government unit. It turns out that after implementing the regulation (and in consequence of establishing SEZ) the employment level has increased to Y1 (point B Chart 2). Does that mean (in both cases) that the difference between Y1 and Y0 equalizes the impact (effects) of our regulation? Y1-Y0= intervention impact?

As mentioned above, adopting such an assumption would be extremely risky because we realize that the enterprise revenues or the employment level in the municipality are influenced not only by the public intervention. The impact can be also exerted by a number of factors such as: the economic situation, the

104

situation in a given branch, changes in the state fiscal policy or at last a natural development trend for supported companies/the whole economy. Therefore it is necessary to assume that the enterprise revenues or the unemployment level would have changed one way or another, even without the support offered within the economic zone (see Diagram 6.) Diagram 6. Intervention impact (2)

Intervention effects B

Observation

Counterfactual situation

1-0 implementation of regulation before implementation

1-1

time (x)

B Observation

Counterfactual situation

after implementation of regulation

Employment level in LGU

Average annual enterprise revenue

1-0 implementation of regulation before implementation

1-1

time (x)

after implementation of regulation

Source: by authors

In view of the above, while assessing the intervention impact it is necessary to make an attempt of answering the question what the situation in the analyzed area (influenced by our intervention) would look like if no regulation had been implemented at all. In the literature this hypothetical situation is referred to as counterfactual. We cannot observe it in reality (because we do not have at our disposal a group of enterprises/municipalities which at the same time are and are not the support addressees). The main idea (and at the same time the main difficulty) of conducting research based on counterfactual methods just comes down to precise defining what situation the addressees of support would be in if they had not obtained the offered support. After we have estimated what level of the enterprise revenue (or the employment level in case b) would be in this counterfactual situation, defining the real impact is just a sheer formality. It is the difference: Y1- Y1* (see Diagram 7.)

105

Diagram 7. Intervention impact (3) Intervention effects B

Observation Impact=Y1-Y2 Counterfactual situation

1-0 implementation of regulation before implementation

1-1

time (x)

B Observation Impact=Y1-Y2 Counterfactual situation

after implementation of regulation

Employment level in LGU

Average annual enterprise revenue

1-0 implementation of regulation before implementation

1-1

time (x)

after implementation of regulation

Source: by authors

The research based on counterfactual methods makes it possible to recreate approximately (if the assumptions are relevant) such a ‘virtual’ reality. It is conducted by estimating a hypothetical situation, usually thanks to properly selected control groups (comparative groups). It is essential that this group should be possibly most similar to the intervention group (support addressees). The only characteristic which makes them different from the group of beneficiaries is that they have not obtained the support. How to make up a virtual reality (selecting the proper control (comparative) group)? In practice the two following approaches can be used: • experimental • quasi-experimental.

3.6.1. Experimental approach

Experiment is an intervention in which a researcher makes a planned change in some factors related to the researched situation and at the same time controls other factors in order to obtain, by observation, an answer to the question about the results of this change (Sułek, 1979). The selection of both groups is made at random. The situation we have to do is the so-called ‘golden standard’ (Rossi et al., 1999) In the most general way the progress of experiment consists of the following stages:

106

1. Out of an accessible population (defined by the planned scope of a given intervention) is selected a smaller sample (usually the voluntary criterion is used- individuals apply for this intervention on their own will at this stage). 2. Then the selected sample is divided into an experimental group (subject to the intervention) and a control group (without the support). The division into the two groups is done at random. 3. The experimental group takes part in the analyzed intervention. The control group is monitored. 4. After the implementation has been completed, the intervention impact is being analyzed by comparing the experimental group with the control group. Possible differences are interpreted in terms of causal impact of a given intervention. The random division of units into experimental and control ones is a guarantee of comparability of the two groups in statistical terms. This fact makes it theoretically (i.e. assuming that the experiment has been conducted correctly) possible to identify the cause-and-effect relationship between the implemented initiative (intervention) and the observed changes. Thanks to it, experimental methods are potentially the strongest bases of estimating the real impact of public interventions. Despite the fact that the experimental method provides really sound evidence as for the effectiveness of public initiatives, it is not very often used. The reasons for this are as follows: • Ethical problems (especially when a given intervention could bring negative outcomes – e.g. driving without safety belts, smoking cigarettes, etc.) • Political correctness (an intervention is conducted in a one-sided way which favours evaluation, the evaluator decides who will obtain the support) • The research is costly • The research takes a lot of time and there is a risk of obtaining out-of-date results • It is impossible to decide on conducting the research when the intervention is in progress (‘disruptions’ of a potential control group)

107

Nowadays experimental methods are relatively often used in developing countries, especially by large international institutions (such as the World Bank). They are also very popular in the USA where they complement pilot schemes prior to implementing public interventions on a large scale. However, it should be stated that even by using experimental methods it is not possible to avoid errors in the research. Moreover, it is necessary to remember that the experiment itself can disrupt the environment of the population.

3.6.2. Quasi-experimental methods

In view of the specificity of public interventions implemented in the EU countries quasi-experimental methods are used more frequently. Some of them are presented below. Difference-in-difference (DID) The main assumption of the technique is that potential intervention impact (e.g. as for the employment level) is subject to a kind of dynamics in time as for the support target group as well as the group without the support. The truth is that we do not know how the situation of the entities being subject to a given regulation would be changing if this regulation didnâ&#x20AC;&#x2122;t concern them (i.e. in the counterfactual situation), but it can be assumed that its good equivalent is related to the growth trend observed in a group of entities which are not subject to the impact of a given intervention. Let take us a look at the example shown in the diagram below. In this case we can imagine that the companies located in a special economic zone have increased their R&D expenditures19 from PLN 65,000 to PLN 75, 000 (purple line). We know, as it was stated above, that the simple comparisons of the state before and after could fake substantially the reality (the impact valued at PLN 10.000). We guess that even without the support, the SEZ companies would be very likely to change their average level of R&D expenditures. Adopting the DID approach we assume that the enterprises taking advantage of the support, which is a location in a SEZ, would be developing just like all other companies (in a given region) if they had not obtained this support. The pace of increasing R&D 19

e treat the increased R+D expenditures as one of the possible effects of a company localization W in SEZ.

108

expenditures for all enterprises in the region is shown as the green line (this information can be obtained from public statistics). In practice assessing the support impact involves comparing the situation of public intervention beneficiaries with the change observed at the same time in the selected comparative group. This can be expressed in the following way: Intervention impact=(Y1,AFTER – Y1,BEFORE) – (Y0, AFTER-Y0, BEFORE)

In the example below the impact equalizes the observed difference in time between R&D expenditures in the group being subject to the regulation (PLN 15,000 growth) and the comparative group (PLN 10,000 growth). In this situation the assessed intervention impact amounts to PLN 5,000. Diagram 8.

T=0

Y (AVERAGE R&D EXPENDITURES)

80 000

75 000

70 000

70 000 65 000

65 000

60 000 55 000

T=1

Impact= dofference after – dofference before = 15 000 – 10 000 = 5 000

60 000 55 000

50 000 PRE

POST

Source: by authors

At this point we have to clearly define assumptions which must be fulfilled in order that the indicated PLN 5,000 could be actually identified with the support impact. These are as follows: 1. The counterfactual (unobserved in the support group) trend is parallel to the control group trend – in other words, the control group trend is a good equivalent to what the situation of the supported entities would be

109

like if they had not obtained the support (they had not been subject to the intervention) 2. All unobserved differences between beneficiaries and the control group are invariable in time 3. Potential selective encumbrance has an influence on the starting point (initial differences between the groups), not on the trend Fulfilling the above assumptions in practice will depend on a number of factors, including mostly the way of selecting the comparative group. Obviously, the way of selecting the control group in DID is an open matter – starting with possibly the simplest situation in which a group of participants of a given intervention is compared to all entities which are beyond its impact. It is also likely that the companies taking advantage of the R&D support are on a different development path from the control group. The truth is that the assumption of the parallel trend may be verified. For this , however, two additional points in time are necessary (before providing the support). In many cases the compared populations will vary so much that their development trend will be completely different. In view of this, other methods complementary to DID are used. They involve controlling significant characteristics in comparable groups in the way which guarantees lack of substantial differences in observed variables. Such techniques are generally based on matched samples. The text below describes their most important characteristics.

More on DID, see: • Angrist J., Pischke J.S. [2008], Mostly Harmless Econometrics, Princeton University Press, NJ • Bertrand M., Duflo E. and Mullainathan S. [2004], How Much Should We Trust Differences-in-Differences Estimates?, in: The Quarterly Journal of Economics, 2004, vol. 119, n. 1, pp. 249-275. • Card D., Krueger A. [1994], Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania, w: The American Economic Review, 1994, vol. 84, n. 4, pp. 772-793. • Card D., Krueger A. [1997], Myth and Measurement: The New Economics of the Minimum Wage, Princeton, NJ, Princeton University Press.

110

â&#x20AC;˘ ITPS [2004], The EC Regional Structural Funds impact in Sweden 19951999:A quantitative analysis, Swedish Institute for Growth Policy Studies, University of UmeĂĽ, Department of Geography.

Techniques based on matched sample As for the techniques based on matched samples, our efforts are made to create a control group in such a way that it will be similar to the support group (subject to intervention) as much as possible in defined observable characteristics. The only important factor which should differ the support addressees from the selected control group is the obtained support. This approach assumes that all significant differences between the group which has taken part in the programme and the group beyond the intervention can be totally explained in terms of observable characteristics (Bryson et al., 2002). In general, these techniques try imitating the experiment and to be precise, matching individuals on the basis of the value of control variables is to play an analogous role to the random sampling mechanism in the experimental method. It is assumed conditionally in view of the observed (control) variables that the process of matching observation and the intervention group is random as for the estimated impact variable Y (Byrson et al., 2002). The correctness core of the above logic is a proper selection of controlled characteristics which must have an actual influence on the causes of the obtained support on one hand, and on its estimated results on the other hand. From a technical point of view, matching a control group can be exact (control units are identical to entities in the intervention group) or approximate (possible some differences within single units, observed distributions of the control variables at the level of comparable groups are minimized). Such situations are described below. Exact matching In the case of a small number of observed characteristics we can try to match identical units in pairs. In practice such an approach is translated into matching each unit being subject to a regulation (e.g. enterprises located in SEZ) to at least one unit from a control pool (enterprises located outside SEZ). In practice, the described approaches encounter one serious problem, which

111

is difficulty in matching the control group, which becomes bigger along with a number of control variables. In view of assumption fulfillment, it is relevant that the controlled range of characteristics should be as wide as possible. Unfortunately, the more the controlled variables, the slimmer chance of finding identical units and matching them in pairs. In practice matching the control group through exact matching requires relatively extensive sets of data (as compared to the group size of support beneficiaries). The matching problem is multidimensional and it may not be solved in reality. In view of this, in the 80s of the 20th century other derivative techniques were developed. They allow for taking account of a larger number of characteristics within the same assumptions. One of them is propensity score matching.

Propensity score matching The creators of the technique called propensity score matching are Rosenbaum and Rubin who described it in their article20 in 1983. They presented solving a problem related to difficulty in controlling many different characteristics by bringing them all to one synthetical indicator called propensity score. It can be defined as the support participation probability (it has scores 0 and 1) while taking into consideration observed characteristics of the support participants. The key condition for using the technique is a proper set of data concerning both regulation addressees and entities beyond the regulation. These data must refer to the period as early as before starting the intervention. The procedure of conducting the research based on propensity score matching is as follows. The first step is to estimate propensity score for each support addressee as well as each entity beyond the support. Propensity score is calculated on the basis of prior defined characteristics which, in the researcherâ&#x20AC;&#x2122;s opinion, have an influence on participating a unit in the support and thus on the obtained impact. The second step is matching units of the experimental group and the control group. Different matching methods can be used. The selection of the proper one 20

osenbaum P.R., Rubin D.B., [1983], The central Role of the Propensity Score in Observational Studies R for Causal Effects, Biometrica, Vol.70

112

is quite difficult and each decision brings certain consequences for the research results (more on the subject in: Rosenbaum and Rubin, 1983, pp. 43-44). After the two groups are identified, the third step is made to assess the quality of matching the comparable groups. It is necessary to check whether the matching procedure has made it possible for us to balance the distributions of the variables introduced into the model in the control group and the experimental group. It can be done by visual comparing the frequency of propensity score distributions in both groups (Lechner, 2002, pp. 59-82) and analyzing standardized differences in the values of the controlled characteristics (more on the subject in: Trzciński, 2009). The fourth step is to calculate the support impact. We are in the best situation when we possess information on the impact indicator also before starting the intervention with reference to both regulation addressees and entities which are not subject to the regulation. If so, we can additionally use difference –in-differences. Methods based on propensity score matching can provide relatively sound evidence as for analyses of public support impact. The necessary condition for using the technique is possessing a wide range of data concerning both support addressees and entities beyond the support. These data must be accessible before starting a given intervention as well as after its completion. In view of this it is extremely important to design how the impact of the intervention will be quantified before the intervention starts. More on this research market, see: • Angrist J, Pischke J.S. [2008], Mostly Harmless Econometrics, Princeton University Press, NJ. • Bryson A., Dorsett R. and Purdon S. [2002], The Use of Propensity Score Matching in the Evaluation of Active Labour Market Policies, UK Department for Work and Pensions, London. • Caliendo M., Kopeinig S. [2008], Some Practical Guidance for the Implementation of Propensity Score Matching, Journal of Economic Surveys, 22(1), pp. 31-72. • Cochrane W., Chambers S. [1965], The Planning of Observational Studies of Human Populations, in Journal of the Royal Statistical Society, Series A, 128, pp. 234-266.

113

• Gadd H., Hansson G. and Månsson J. [2008], Evaluating the Impact of Firm Subsidy Using a Multilevel Propensity Score Approach, Centre for Labour Market Policy Research (CAFO), School of Management and Economics, Växjö University. • Hirano K., Imbens G. [2004], The Propensity Score with Continuous Treatments, in Missing Data and Bayesian Methods in Practice: Contributions by Donald Rubin's Statistical Family, Wiley, NYC. • Imbens G. [2004], Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review, in: The Review of Economics and Statistics, 86(1), pp. 4-29. • Lechner M. [2002], Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods, in Journal of the Royal Statistical Society, Series A, 165, pp. 59-82. • Rosenbaum P., Rubin D. [1983], The Central Role of the Propensity Score in Observational Studies for Causal Effects, in: Biometrika, 70, pp. 41-50. • Strawiński P. [2008]. Quasi-eksperymentalne metody ewaluacji, w: Haber A., Szałaj M. [red.], Środowisko i warsztat ewaluacji, PARP, Warsaw • R. Trzciński [2009], Wykorzystanie techniki propensity score matching w badaniach ewaluacyjnych, PARP, Warsaw.

Discontinuity design This technique can be used in the situation when obtaining the support is determined by the rule stating that below/above a certain characteristic value (continuous characteristic- e.g. enterprise size, child age, a number of children, household income) a given group is or is not subject to a regulation. In view of this the control group and the comparative group are composed of units positioned nearby the defined borderline. The situation is quite similar to the experiment based on random sampling as obtaining the support is determined by randomness (of finding oneself on the one or on the other side of the borderline). In this case analyzing the support impact comes down to comparing units positioned on both sides of the borderline within the defined tolerances. The method provides very strong arguments as for assessing the impact on the researched group. The problem may arise when we are aware that functioning units in the two groups changes substantially along with moving from the

114

defined ‘border’. It is possible then to precisely define only the impact on the units staying very near this border. At the same time we have no right to generalize the results with reference to the whole researched population. In this case, the technique can be combined with DID as well. More on the research technique see: • Battistini E. Rettore E [2008] 'Ineligibles and eligible non participants as a double comparison group in a regression discontinuity design', «Journal of Econometrics, vol. 142, n. 2,pp. 611-850 • Cook T. [2008], Waiting for Life to Arrive: a History of the Regression-discontinuity Design in Psychology, Statistics and Economics, in: «Journal of Econometrics, vol. 142, n. 2, pp. 636-654. • Hahn J., Todd P., Van der Klaauw W. [2001], Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design, in: «Econometrica, vol. 69, n. 1, pp. 201-209. • Serrano-Velarde N. [2008], The Financing Structure of Corporate R&D Evidence from Regression Discontinuity Design, Working Paper, European University Institute, Department of Economics, Firenze. • Thistlewaite D.L., Campbell D.T. [1960], Regression-discontinuity Analysis: An Alternative to the Ex-post Facto Experiment, in: «Journal of Educational Psychology, n. 51, pp. 309-317. • Trochim W. [1984], Research Design for Program Evaluation: The Regression-discontinuity Approach, Beverly Hills, CA, Sage Publications.

3.5.3. When can we use methods based on counterfactual approach?

Using counterfactual methods, it should be remembered that they are not always the most appropriate for defining the impact of implemented interventions. Counterfactual methods fit in to a situation when it is possible to match ‘twins’ which are not subject to a regulation. As for drafted legal acts, we have to do most frequently with a situation when all entities (entrepreneurs, tax payers, LGUs) are subject to a regulation, but it is not always like this. Some regulations provide an opportunity of obtaining the support, however it is not a rule. This is the case of the above presented example concerning special economic zones. Municipalities or local governments are not obliged to take advantage of the support. Another example which

115

does not refer to the support but to a certain obligation imposed on society can be the situation related to sending six- or seven-year-olds to schools (some time ago a hot subject in Poland). The legislator left a choice to be made for parents. We have to do with a different situation when we assess the impact of regulations being the basis for putting financial support instruments (e.g. programmes) into operation. Sometimes such instruments are defined explicitly in the legal act itself. In this respect evaluation experiences would be invaluable as reference points. The basic criteria which should be taken into consideration, while deciding whether we can adopt a counterfactual approach, are as follows: a large number of support addressees, repetitiveness and homogeneity of the support. A synthetical tabulation of interventions undertaken within cohesion policy and with reference to the relevance of using counterfactual research for the impact quantification is presented in the table below. Table 5. Types of intervention versus application of counterfactual methods Different types of investments within Cohesion Policy R&D support

Company grants

Renewable sources of energy

Revitalization

Transport infrastructure

Human capital investments

Possibility of using impact evaluation

High

Medium

Low

High

Repetitiveness

Large numbers of observations

Support homogeneity

Source: Martini, 2009.

Other interventions are implemented in stages. It is known, e.g. that due to the state budgetary shortages it is not possible to immediately implement a given solution with reference to all entities within one year, so this process progresses within a few years. In this case the control group should include entities matched from a group anticipating the support in order to compare this group with units which have already become subject to the regulation.

116

One more type of intervention can be conducted at a local level (cities/LGU). While assessing ex post a law which leaves cities/LGU a free choice as for different solutions, it is advisable to conduct a further analysis of the impact of adopting different alternatives implemented by different cities/LGU. It should be noted that there has been a systematic increase in the high level decision-makersâ&#x20AC;&#x2122; consciousness of developing evidence-based policies. Its natural consequence should be enforcing particularly important or risky legislative solutions within a pilot schemes, which later ought to be followed by a reliable analysis of the implemented solution. Such a pilot scheme would be a condition for spreading the intervention over the whole population. This practice is used e.g. in the USA. Also, some countries of Western Europe have applied such solutions (the first â&#x20AC;&#x2DC;guinea pigsâ&#x20AC;&#x2122; are often officials of the institution responsible for enforcing a regulation, who personally test new solutions). In the case of a pilot regulation it is advisable to conduct experiment-based research. To sum up, it is necessary to be aware that counterfactual methods are well applicable only in assessing some types of intervention. They allow to precisely define relatively homogenous impact of a regulation (or part of them) at the level of particular components, not with reference to extensive, multidimensional legal acts. Thanks to these methods we are able to define the impact scope, however, we cannot obtain an answer why the impact has occurred. Finally, using methods based on a counterfactual scheme, we can precisely assess the expected impact of our regulation but at the same time we lose sight of all unexpected effects which may be important. In view of the above, is it worth making an effort of conducting research according to a design based on these methods? Undoubtedly, the answer is yes! Experiences in conducting Cohesion Policy evaluations have definitely confirmed great utility of this research. Very often their results have provided key arguments in discussion on reorientation of financial initiatives within EU resources. This research has contributed not only to greater effectiveness of European funds, but also enabled diagnosing the situations in which, despite good intentions, we have done harm to programme beneficiaries instead of helping them. In recent years the European Commission has also put a lot of emphasis on using these methods, not only in view of Cohesion Policy but also of regulatory initiatives which are related, among others, to the research on the impact of public aid provided to enterprises.

117

Appendix 1. Contracting research. Description of conditions related to contracting additional research for RIA While preparing regulatory impact assessment it may be necessary to contract additional research. Such a decision could result from the following: • Lack of enough knowledge (methodological or sectoral) possessed by a unit developing RIA • Necessity of conducting research by external experts who are authorities in their area • Lack of time or technical resources (e.g. a studio for telephone research) necessary for conducting research in-house (Olejniczak et al., 2012). Identifying the main causes of contracting research will help prepare a proper description of the contract subject and the selection criteria thanks to which there is a greater chance of selecting the best offer. If the decision about contracting research has just been made it is necessary to take subsequent steps in the research contracting process. These steps are presented in the diagram below and then discussed in detail. Diagram 9. Research contracting process

Research design (contract description)

Assessing the contract value

Selecting contract mode

Defining conditions of participation in proceedings and criteria of assessing tenders

Preparing final version of contract documents

Source: by authors

The first step is to prepare a research design which should include the following information as the minimum: • Research objective • Justification of conducting research, its recipients and planned application of the results • Research background

118

• Research scope • Research questions • Research methodology • Research schedule • Accessible information sources • Requirements of result presentation If research is contracted, its design will include a description of the contract subject (DCS). As experiences in evaluation research confirm, there are different approaches to describing the contract subject. Some Clients prepare a very detailed DCS which contains an extensive description of the research scope and questions as well as its methodology. Other Clients, in turn, describe in detail the research objective, defining only methodological minimum which must be fulfilled by their Contractor. The selection of the approach depends on the one hand on underlying causes of making a decision about outsourcing research. One approach (detailed DCS) is adopted by units in which the decision about outsourcing research has been taken due to lack of time, human or technical resources necessary to conduct the research in-house. Another approach (methodological minimum in DCS) is used by units deficient in methodological or sectoral knowledge necessary for conducting the research in-house and because of this also unable to describe the research in DCS. On the other hand, the same approach can be also used by units which possess sufficient knowledge of preparing DCS but want to select the research Contractor on the basis of the substantive tender assessment. The scope of DCS is also adjusted to the possessed resources such as: • Data (whether necessary data are accessible) • Budget (costs) • Time (preparing tools, making measurements and collecting data, analyzing data and compiling results The contract budget includes the following estimated costs: • Preparing tools and an in-depth methodological concept of the research if DCS is less precise (most frequently the cost is measured by a number of man-days of methodologists and thematic experts) • Conducting measurements and collecting data (the cost of conducting particular research methods and techniques)

119

• Analyzing data (the cost is measured by a number of man-days of statisticians and thematic experts) • Compiling results (the cost is measured by a number of man-days of thematic experts) After preparing the research description and its valuation21 it is possible to select a mode of proceedings. As for research below the defined borderline (in Poland -the value of 30 000 euros) the Public Procurement Law (PPL) is not applicable. However, even in this case it is worth using some elements of the proceedings which are subject to the Law, namely the conditions of participation and the criteria of tender selection which are described below in this Appendix. According to the Polish PPL, the basic modes of the award of contract are open tender and restricted tender. Having complied with all the conditions defined in the law the Client can also award the contract in the following modes: negotiations with the announcement, competitive dialogue, negotiations without the announcement, single source procedure, price enquiry or electronic bidding. The selection of the mode of proceedings of the award of public procurement for research depends especially on: • Possibilities on clarifying DCS • Time for proceedings If the Client’s knowledge and skills are not sufficient enough to prepare precise DCS they can select the mode of negotiations with the announcement (Polish PPL:’ Specific characteristics of the contracted services cannot be defined in advance in a way which facilitates the selection of the most favourable offer in the mode of open tender or restricted tender’) or competitive dialogue (Polish PPL: ‘if total: it is not possible to award the contract in the mode of open tender or restricted tender because on the ground of a particularly complex character of the contract it is not possible to describe the subject of the contract in compliance with Art. 30 and 31 or objectively define legal and financial conditions of the execution of the contract; the price is not the only criterion of selecting the most favourable tender’). In such a case the research 21

he contract value is defined at the earliest 3 months prior to the day of initiating the proceedings T of the award of the contract for services

120

design will be based on the knowledge of the Contractors, however it is a good idea to hire an external expert to support the Client in the negotiations or the dialogue. Using negotiations within the tender selection allows to discuss proposals, explaining doubts, pointing out shortcomings and potential threats as well as the scope of aid from the Client. Such meetings facilitate familiarizing with research teams and getting to know which offer will suit best the Client’s needs. In practice, however, the proceedings within the modes mentioned above require much more time than in the case of the basic modes, i.e. open and restricted tenders. Time is also a factor in favour of the mode of negotiations without the announcement (Polish PPL: ‘ due to an urgent need of the award of the contract for the reasons directly not attributable to the Client but unlikely to have been foreseen, it is not possible to keep the deadlines defined for open tender, restricted tender or negotiations with the announcement’). However, in practice it could be difficult to justify the occurrence of such reasons as for the research necessary for conducting IA. At this point it is also worth mentioning complementary contracts which can be awarded in the mode of single source procedure provided that they: • are included in Terms of Reference (TR) • involve repeating the same types of contracts • are awarded within 3 years from the award of the main contract on • do not exceed 50 % of the value of the main contract. As for IA in practice, we can imagine that the research aimed at diagnosing the situation in a defined area for ex ante RIA has been conducted and it has been repeated for ex post RIA in order to assess regulatory impact (the problem can be time, namely fitting the two attempts in to 3 years). Another important step in proceedings of the award of the contract is to define the conditions of participation and the criteria of tender assessment. These elements are related to each other along with the conditions which determined the decision about contracting the research. The conditions of participation must result from the Client’s needs and be cohesive with the criteria of tender assessment. If the Client does not possess enough knowledge about methodology, the conditions of participation should mostly concern members of the research panel and refer to the experience

121

(possibly to education) in designing and conducting research, using research methods (collecting and analyzing data). In the case of the Client’s insufficient knowledge of the research subject, the conditions of participation should mostly concern members of the research panel and refer to experience (possibly education) in the researched area. The same requirements for members of the research panel (experience in methodology and research area) should be made by the Client when they expect the research to be conducted in an innovative or original way. If the Client is keen on the research conducted by researchers - authorities of renowned names, the conditions of participation in the proceedings to be met by members of the research panel must refer to science degrees, publications as well as knowledge and experience in a defined type of research (Olejniczak et al., 2012). The conditions of participation with regard to members of the research panel can be complemented by requirements for the Contractor (companies) and they can refer to their experience in a defined type of research. As for the criteria of tender assessment, in view of the specificity of the contract subject i.e. the research, whose utility depends mostly on the substantive value of the results, it is recommended to use the proportion in the criteria selection which will ensure selecting the tender of the highest level of substantive value. In view of the above, the score for a methodological approach and a way of organizing and conducting the research (substantive criteria) should make up a significant percentage of all the score to be gained when the tenders are assessed (MRD, 2007). Qualitative criteria of tender assessment e.g. can refer to: • Scientific and research background • Research concept • Organization of the research process. Scores (points) should be ascribed to the criteria (more information in the further text) and subcriteria ought to be defined. Doing the latter should depend on the Client’s capabilities of clarifying DCS of their needs with reference to the quality of the research results. The assessment criteria should concern the issues (research methodology or area) which have not been clarified in the Terms of Reference due to the Client’s insufficient know-

122

ledge22 or their choice to leave this aspect to be clarified by the tenders (a chance of presenting an innovative and original research concept). Below is a catalogue of possible subcriteria23 which refer to the following (Olejniczak et al. 2012): a) scientific and research background: i) relevance of research background (in other words: comprehension of the research subject) and/or scientific theories (supported by references to publications). b) research concept: i) relevance or utility (in terms of the research objectives) of expanding the research concept by research areas or questions , ii) r elevance of clarifying research objectives/areas by proposing adequate indicators, iii) r elevance or utility (in terms of research objectives) of indicating research questions or relevance of expanding the research concept by additional research questions, iv) relevance or utility (in terms of research objectives) or innovativeness or integrity of indicating research methodology or its expanding by additional research methods/techniques, v) cohesion or integrity of the research logic ( in other words: connecting the research areas, cause-and-effect relationship between the research stages or areas, matching research methods to questions and questions to objectives, vi) level of differentiating research methods, vii) level of differentiating data sources, viii) relevance of the way of engaging stakeholders. c) research process organization: 22

his case brings the problem of assessing offers within the criteria referring to the aspects in which T the Client possesses not enough knowledge. It can be solved by contracting an independent expert who will support the substantive assessment of offer within a given scope. In this case it is also worth having them participate in the process of preparing DCS and TR. I n TR in the item concerning requirements for the offer, the Client must indicate all the elements which must be included in the offer in the way making it possible to assess the tender according to the assumed criteria of tender assessment. For example, if among the selected criteria of tender assessment is the criterion concerning relevance of additional research methods, it should be required that tenders should state clearly which methods described in the offer are additional (with reference to those required in DCS).

123

i) relevance of possible identified risk areas which may occur in the research progress and means minimizing these risks, ii) r elevance of means/instruments influencing improvement of research reliability, iii) r elevance or innovativeness of the ways of coordinating and controlling the research quality, iv) utility or innovativeness of solutions related to presenting or publicizing the research results. The criteria can be scored in the three following ways (Olejniczak et al., 2012): • score brackets – ‘up to…….points; the criterion should include a factor assessing a level of e.g. utility, cohesion, relevance, etc. • grading, e.g. ‘1 point for 2 relevant questions, 2 points for 3 questions, 3 points for 4 questions….’; in this case the criterion should also have a factor assessing a level of e.g. utility, cohesion, relevance, etc. • either/or scoring (it either meets or it does not meet the criterion); in this case the criterion should have a form: ‘relevance’, cohesion’ (e.g. '3 points – relevant identification of potential risk area which can occur in the research progress and the means which minimize them; 0 points – potential risk areas and/or means minimizing them have not been identified or some of those identified are not relevant’). The score which differentiates the Contractors most is the first one, i.e. defining a maximum points likely to be scored. The other two scores do not provide tenders so great an opportunity of presenting their concept which proves their competence as for the contract subject. These scores will encourage adjusting the research concept to the criteria of tender assessment and it may turn out that, in fact, they do not differentiate tenders to much extent (all or most tenders will be given the same number of points) and thus the selection of the best substantive offer will be difficult. In view of the specificity of research for IA whose utility depends mostly on the substantive result value it is recommended to use proportions in matching criteria for the selection of the Contractor as they can ensure the selection of the offer of the most substantive value. Given the above, while conducting substantive assessment of the submitted tenders, points given for a methodological

124

approach and for a way of organizing and conducting the research (substantive criteria) should account for a substantial percentage of all the score (in practice substantive criteria should be at the level of 50%-80% of the total score). The last element of the process of contracting the research is verifying the cohesion of all elements of DCS and TR such as: the conditions of participation, criteria of assessing tenders and requirements for the tender.

125

Appendix 2. S pecimen documents to be used in designing and conducting research Specimem evaluation research schedule

Task

START OF RESEARCH Preliminary analysis of available data and developments Opening meeting with the Client METHODOLOGICAL REPORT Methodological report -draft Agreeing on methodology with the Client Preparing and presenting draft research design Methodological report- ultimate version RESEARCH AND ANALYTICAL WORKS e.g.: Desk research: IDI FGI Method x Workshops with the Client FINAL REPORT – DRAFT FINAL REPORT – ULTIMATE VERSION ADVISORY SUPPOADVISORY SUPPORT FOR THE CLIENT

Research panel participants

date week I

III

VI VII VIII IX

XII

126 Specimen letter of reference

LETTER OF REFERENCE Dear Sir or Madam, I would kindly like to inform that ……………………… has started the evaluation research on …………………………………………. The research will be conducted by the independent Evaluator - ………………………………………..company. The main objective of the research is a complex assessment of the relevance and cohesion of ………………… and an assessment of the system……………………………, including human and administrative resources, as well as drawing conclusions and recommendations on changes in……………………….. . The scope of the evaluation research includes among others: a) Analysis of materials: scientific and evaluation research as well as community and national strategic documents applicable in evaluation b) Group interviews with thematic experts as well as social and economic partners c) Personal in-depth interviews with representatives :…………………………… d) Telephone interviews of………………………………………………….. e) Workshops with representatives of…………………………………………. f) Research ……………………………………………………………… The Research Panel consists of: ………………………………………………………… The research will involve participatory evaluation, i.e. the Client will acompany ………………. at all key stages of preparing……………………………………… . I would like to ensure that the members of the Panel will be guided by the principle of statistical confidentiality in their course of action and that they will collect only the information which is closely connected to the research. Hence I would sincerely ask you to cooperate and give the members of the Research Panel any help necessary for undertaking their tasks as well as provide them with access to the information related to the research. I would like to point out that your assistance in conducting the research is a necessary condition for its effectiveness and the utility of the result research o be further used in works on……………………… Should you need any further information on the subject research, please contact: - on behalf of the Client - on behalf of the Contractor Yours faithfully, Source: Bienias et al., 2012, p. 67.

127 Declaration of Confidentiality

DECLARATION OF EVALUATOR’S CONFIDENTIALITY Ref.: The research on. ...................................................................................................................................... I hereby agree to respect confidentiality and secrecy with regard to any information or documents submitted by the Client or discovered /prepared by me in the course of and in relation to conducting the evaluation research on. ................................................................. I agree that the information be only used for the purpose of subject evaluation and that it not be revealed to the third parties. I also declare not to keep the copies of the information provided with reference to the copyright in particular. I take responsibility of returning all the submitted copies after they have been used by me for conducting the above mentioned evaluation. Name and surname: .................................. Signature: ............................................... Date: .................................................. Source: by authors based on: Bienias et al., 2012, pp. 66.

Specimen final report structure A final report, as the final product of the impact assessment of a draft legal act, should extensively refer to all evaluation questions. The main content should not be included in more than 100 pages, excluding a summary and appendixes. The researchers should hand in the final report in its electronic and paper forms (X copies). The report ought to be prepared in a national language (and in English in its ultimate version). The structure of the final report is supposed to reflect specific objectives of the research and its layout is to be based on the following elements: • One-page resume • Abstract up to 5 pages (A4) – it allows the reader to use the material on their own and it should include the most significant results of the whole research, including visual prompts • Table of contents • Introduction including a description of the subject , the main research assumptions and objectives as well as the research circumstances • Synthetic description of used methodologies

128

•

Description of the research results with reference to particular research questions, including a general outline of assessing a draft legal act (analytical sections should include additional synthetic summaries) as a separate section of the report • Preliminary concept of evaluation process for a draft legal act • Conclusions (summing up the research with reference to the specificity of the research area) and recommendations related to them. Appendixes, including the scope description and the way of adopting conclusions and recommendations implemented in the research process (changes/ improvements) Source: by authors based on :Bienias et al., 2012, pp. 69-71.

Specimen final report assessment form Research title: ................................................................................. Criteria of evaluation research assessment:

Insufficient

Poor

Sufficient

Good

RELEVANCE Does the content correspond with identified informative needs?

TAKING ACCOUNT OF THE STATE OF SCIENTIFIC KNOWLEDGE Has the executor referred to the existent state of knowledge regarding the research subject in the research process (at all stages)?

METHODOLOGY USED Have the methodology and the way of conducting the research (proposed methods and tools) been appropriate for achieving the research objectives and obtaining answers to the evaluation questions?

DATA RELIABILITY Are the collected data adequate for the designed application and has their reliability been checked?

QUALITY OF ANALYSES AND CONCLUSIONS Are quantitative and qualitative data analyzed in the correct way?

REPORT TRANSPARENCY Is the report well-structured, correctly composed (harmonious) and written clearly and comprehensibly?

REPORT SUMMARY ASSESSMENT If the report summary written synthetically, clearly and comprehensibly?

Excellent

129

RECOMMENDATION ASSESSMENT Are the formulated recommendations clear, useful and objective?

RECOMMENDATION TABLE ASSESSMENT Has the table been worked out synthetically, clear and precisely?

ASSESSMENT OF COOPERATION WITH RESEARCH CONTRACTOR Matters related to keeping in touch in the research 10 progress, consulting and taking account of the Clientâ&#x20AC;&#x2122;s remarks, keeping deadlines, flexibility in the case of unexpected background factors

ASSESSMENT OF COOPERATION WITH OTHER ENTITIES IN THE RESEARCH PROGRESS 11 What has the cooperation of the Client with other external entities (depending on the research specificity , e/g/ key stakeholders) been like?

Source: based on the material provided by the National Evaluation Unit

131

Bibliography 1. Angrist J., Pischke J.S. [2008], Mostly Harmless Econometrics, Princeton University Press, NJ. 2. Babbie E. [2008], Badania społeczne w praktyce, Wydawnictwo Naukowe PWN, Warszawa. 3. Babbie E. [2013], Podstawy badań społecznych, Wydawnictwo Naukowe PWN, Warszawa. 4. Bamberger M. [2009], Institutionalizing impact evaluation within the framework of a monitoring and evaluation system, IEG World Bank. 5. Bardach E. [2007], Praktyczny poradnik do analizy polityk publicznych. Osiem etapów skutecznego rozwiązywania problemu, Małopolska Szkoła Administracji Publicznej [with approval of CQ Press], Kraków. 6. Battistini E., Rettore E. [2008], Ineligibles and eligible non participants as a double comparison group in a regression discontinuity design, Journal of Econometrics, Vol. 142, n. 2. 7. Batorski D., Olcoń M. [2006], Prowadzenie badań przez internet: Podstawowe zagadnienia metodologiczne. Studia Socjologiczne, 3 [182], Warszawa. 8. Bertrand M., Duflo E., Mullainathan S. [2004], How Much Should We Trust Differences-in-Differences Estimates?, The Quarterly Journal of Economics, Vol. 119, n. 1. 9. Bienias, S. [2010], Ewaluacja efektów netto interwencji publicznych, prezentacja presentation at Warsaw School of Economics. 10. Bienias S., Felcis W., Hermann-Pawłowska K., Trzciński R. [in print], Inspiracje z ewaluacji – najlepsze praktyki, in: Górniak J. [ed.], Ocena wpływu oparta na dowodach. Model dla Polski. 11. Bienias S., Strzęboszewski P., Opałka E. [ed.] [2012], Ewaluacja. Poradnik dla pracowników administracji publicznej, Ministry of Regional Development, Warszawa. 12. Blalock H.M. [1977], Statystyka dla socjologów, Wydawnictwo Naukowe PWN, Warszawa. 13. Bończak-Kucharczyk E. [2008], Strategie rozwoju regionu i miast. Metodologia i wskazówki praktyczne, Minigo.

132

14. Bryson A., Dorsett R., Purdon S. [2002], The Use of Propensity Score Matching in the Evaluation of Active Labour Market Policies, UK Department for Work and Pensions, London. 15. Buzan R., Buzan T. [2003], Mapy Twoich myśli, Łódź. 16. Caliendo M., Kopeinig S. [2008], Some Practical Guidance for the Implementation of Propensity Score Matching, Journal of Economic Surveys, 22(1). 17. Card D., Krueger A. [1994], Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania, The American Economic Review, 1994, Vol. 84, n. 4. 18. Card D., Krueger A. [1997], Myth and Measurement: The New Economics of the Minimum Wage, Princeton, NJ, Princeton University Press. 19. Chen H.T., Rossi P. H. [1980], The multi-goal, theory-driven approach to evaluation: A model linking basic and applied social science. Social Forces, 59. 20. CIiPKZ w Tarnowie, Informacja o zawodzie. Tajemniczy klient http://wup-krakow.pl/uslugirynku-pracy/poradnictwo-zawodowe/informacje-zawodowe-1/o-zawodachpliki/ tajemniczyklient.pdf 21. Cochrane W., Chambers S. [1965], The Planning of Observational Studies of Human Populations, Journal of the Royal Statistical Society, Series A, 128. 22. Cook T. [2008], Waiting for Life to Arrive: a History of the Regression-discontinuity Design in Psychology, Statistics and Economics, in: Journal of Econometrics, Vol. 142, n. 2. 23. Dziemianowicz W., Szmigiel-Rawska K., Nowicka P., Dąbrowska A. [2012], Planowanie strategiczne. Poradnik dla pracowników administracji publicznej, Ministry of Regional Development, Warszawa. 24. European Commission, DG REGIO, Evalsed Sourcebook: Method and techniques, dostępne na: http://ec.europa.eu/regional_policy/sources/docgener/ evaluation/guide/evaluation_source book.pdf 25. European Commission, DG REGIO [2014], Guide to cost-benefit analysis of investment projects, dostępne na: http://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/cba_guide.pdf 26. Flyvbjerg B. [2006], Five Misunderstandings about Case Study Research, Sage Publications. 27. Frankfort-Nachmias Ch., Nachmias D. [2001], Metody badawcze w naukach społecznych, Zysk i S-ka, Poznań.

133

28. IDEA for Development Fountation commissioned by the Ministry of Foreign Affairs [2014], Raport końcowy zbadanie wybranych inicjatyw zrealizowanych w ramach polskiej współpracy rozwojowej udzielanej za pośrednictwem MSZ RP w latach 2011–2013: projekty wspierające rozwój małej i średniej przedsiębiorczości w Gruzji, Warszawa. 29. Gadd H., Hansson G., Månsson J. [2008], Evaluating the Impact of Firm Subsidy Using a Multilevel Propensity Score Approach, Centre for Labour Market Policy Research (CAFO), School of Management and Economics, Växjö University. 30. Gaffey V. [2009], Case studies in the framework of ex post evaluation, 2000– 2006: expectations and experiences of the DG for Regional Policy, Brussels. 31. Góralski A. [1989], Twórcze rozwiązywanie zadań, Warszawa. 32. Górniak J. [2007], Ewaluacja w cyklu polityk publicznych, in: Mazur S. [ed.], Ewaluacja funduszy strukturalnych – perspektywa regionalna. Kraków: Uniwersytet Ekonomiczny w Krakowie, Małopolska Szkoła Administracji Publicznej. 33. Górniak J. [ed.] [in print], Ocena wpływu oparta na dowodach. Model dla Polski. 34. Górniak J., Żmuda G., Prokopowicz P. [in print], Ocena wpływu w kontekście polityk publicznych, w: Górniak J. [ed.], Ocena wpływu oparta na dowodach. Model dla Polski. 35. Haber A., Szałaj M. [ed.] [2007], Ewaluacja ex post. Teoria i praktyka badawcza, PARP, Warszawa. 36. Haber A., Szałaj M. [ed.] [2009], Ewaluacja wobec wyzwań stojących przed sektorem finansów publicznych, PARP, Warszawa. 37. Hahn J., Todd P., Van der Klaauw W. [2001], Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design, Econometrica, Vol. 69, n. 1. 38. Hirano K., Imbens G. [2004], The Propensity Score with Continuous Treatments, in Missing Data and Bayesian Methods in Practice: Contributions by Donald Rubin's Statistical Family, Wiley, NYC. 39. Imbens G. [2004], Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review, The Review of Economics and Statistics, 86(1).

134

40. ITPS [2004], The EC Regional Structural Funds impact in Sweden 1995-1999: A quantitative analysis, Swedish Institute for Growth Policy Studies, University of Umeå, Department of Geography. 41. Journal of Behavioral Decision Making, http://eu.wiley.com/WileyCDA/ WileyTitle/productCd-BDM.html 42. Kierzkowski T. [2002], Ocena [ewaluacja] programów i projektów o charakterze społeczno-gospodarczym w kontekście przystąpienia Polski do Unii Europejskiej, Warszawa. 43. Kluj Ł. [2011], Prezentacja strategii, w: Jak pisać [i prezentować] strategię komunikacji, dostępne na: http://www.instytut.com.pl/sites/default/files/ Jak_pisac_strategie_komunikacji_-_przewodnik.pdf 44. Komendzińska A., Jak napisać notatkę prasową, dostępne na: http://notkaprasowa.pl/poradnik.html 45. Konarski R., Kotnarowski M. [2007]. Zastosowanie metody propensity score matching w ewaluacji ex post. in: A. Haber. [ed.], Ewaluacja ex post. Teoria i praktyka badawcza, PARP, Warszawa. 46. Lechner M. [2002], Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods, Journal of the Royal Statistical Society, Series A, 165. 47. Ledzion B., Olejniczak K., Wojtowicz D., Metodyka badania. Ewaluacja ex-ante ustawy o zmianie ustawy o bezpieczeństwie żywności i żywienia z dnia z dnia 25 sierpnia 2006 r. [methodological report prepared within MORE project]. 48. Leeuw F. [2003], Reconstructing Program Theories: Methods Available and Problems to be Solved, American Journal of Evaluation, n. 24[1]. 49. Leeuw F. [2012], Theory-based evaluation. EVALSED. 50. Leeuw F. [2012], Linking theory-based evaluation and contribution analysis: Three problems and a few solutions, Evaluation 18. 51. Lissowski G., Haman J., Jasiński M. [2008]. Podstawy statystyki dla socjologów, Wydawnictwo Naukowe Scholar, Warszawa. 52. Maison D. [2010], Jakościowe metody badań marketingowych. Jak zrozumieć konsumenta, Wydawnictwo Naukowe PWN, Warszawa. 53. Maison D., Noga-Bogomilski A. [ed.] [2007], Badania marketingowe. Od teorii do praktyki, GWP, Gdańsk.

135

54. Martini A. [2011], Wprowadzenie do podstawowych pytań stawianych w ewaluacji oddziaływania interwencji: Czy interwencja coś zmieniła? A jeśli tak, to dla kogo?, in: Haber A., Trzciński R. [ed.], Ocena wpływu i prognozowanie efektów w badaniach ewaluacyjnych, PARP, Warszawa. 55. Martini A. [2011], Perspectives and limitations of counterfactual situation analysis in complex public interventions, in: Olejniczak K., Kozak M., Bienias S. [ed.], Evaluating the effects of regional interventions A look beyond current Structural Funds' practice, Ministry of Regional Development, Warszawa. 56. Martini A. [2009], VI th European Evaluation Conference, Warszawa. 57. Martyniak Z. [1999], Metody organizacji i zarządzania, Kraków. 58. Maziarz M., Piekot T., Poprawa M., Broda B., Radziszewski A., Zarzeczny G. [2012], Język raportów ewaluacyjnych, Warszawa. 59. Ministry of Economy in cooperation with the Chancellery of the Prime Minister [2015], Guidelines for Undertaking Impact Assessment and Public Consultations within the Governmental Legislative Process (as in March 2015). 60. Ministry of Regional Development [2007], Wytyczne nr 6 w zakresie ewaluacji programów operacyjnych na lata 2007–2013. 61. Nast J. [2006], Idea Mapping: How to Access Your Hidden Brain Power, Learn Faster, Remember More and Achieve Success in Business, New Jersey. 62. Nikodemska-Wołowik A. [1999], Jakościowe badania marketingowe, Polskie Wydawnictwo Ekonomiczne, Warszawa. 63. Obłój K. [2007], Strategia organizacji, PWE, Warszawa. 64. Olejniczak K. [red.] [2014], R[ewaluacja] Wiedza w działaniu, PARP, Warszawa. 65. Olejniczak K., Kozak M., Ledzion B. [ed.] [2008], Teoria i praktyka ewaluacji interwencji publicznych. Podręcznik akademicki. Wydawnictwa Akademickie i Profesjonalne, Warszawa. 66. Olejniczak K., Kupiec T., Raimondo E. [2014], Brokerzy wiedzy. Nowe spojrzenie na rolę jednostek ewaluacyjnych, in: Haber A., Olejniczak K. [ed.], R(ewaluacja) Wiedza w działaniu, PARP, Warszawa. 67. Olejniczak K., Ledzion B., Pander W. [2008], Ocena potencjału oraz stanu dotychczasowych prac w ramach jednostek ewaluacyjnych funkcjonujących w Instytucjach Zarządzających oraz Instytucjach Pośredniczących I i II stopnia w ramach NPR 2004–2006 i NSRO 2007–2013, Warszawa.

136

68. Olejniczak K., Mazur S., Weremiuk A., Miedziński M., Ledzion B., Kupiec T. [2012], Realizacja badań ewaluacyjnych w świetle Prawa Zamówień Publicznych, Ministerstwo Rozwoju Regionalnego, Warszawa. 69. Olejniczak K., Śliwowski P. [2014], Nadchodzi rewolucja? Analizy behawioralne w interwencjach publicznych. in: Haber A., Olejniczak K. [ed.], [R]ewaluacja 2. Wiedza w działaniu, PARP, Warszawa. 70. Olejniczak K, Wojtowicz D. [2015], Ku nowej metodyce analiz interwencji publicznych. Spektrum planów badawczych, Management and Business Administration, Central Europe, Vol. 23, n. 2. 71. Pawson R., Tilley N. [1997], Realistic Evaluation, SAGE Publications. 72. Pirog M., Lanlan Xu [2011], Directions in Public Program Evaluation in the US, in: Olejniczak K., Kozak M., Bienias S. [ed.], Evaluating the effects of regional interventions. A look beyond current Structural Funds' practice, Ministry of Regional Development, Warszawa. 73. Pylak K. [red.] [2009], Podręcznik ewaluacji efektów projektów infrastrukturalnych, Ministry of Regional Development, Warszawa. 74. Rosenbaum P.R., Rubin D.B. [1983], The Central Role of the Propensity Score in Observational Studies for Causal Effects, Biometrika, Vol. 70. 75. Serrano-Velarde N. [2008], The Financing Structure of Corporate R&D – Evidence from Regression Discontinuity Design. Working Paper, European University Institute, Department of Economics, Firenze. 76. Shafir E. [red.] [2013]. The Behavioral Foundations of Public Policy, Princeton University Press 77. Silverman D. [2012], Prowadzenie badań jakościowych, Wydawnictwo Naukowe PWN, Warszawa. 78. Silverman D. [2012], Interpretacja danych jakościowych, Wydawnictwo Naukowe PWN, Warszawa. 79. Stake R. E. [1995], The Art of Case Study Research, Thousand Oaks: Sage 80. Strahl D. [red.] [2006], Metody oceny rozwoju regionalnego, Wydawnictwo Akademii Ekonomicznej im. Oskara Langego we Wrocławiu, Wrocław. 81. Strawiński P. [2008]. Quasi-eksperymentalne metody ewaluacji. in: Haber A., Szałaj M. [ed.], Środowisko i warsztat ewaluacji, PARP, Warszawa. 82. Sułek A. [1979], Eksperyment w badaniach społecznych, Wydawnictwo Naukowe PWN, Warszawa.

137

83. Sztabiński, P.B., Sawiński, Z., Sztabiński, F. [2005], Fieldwork jest sztuką. Wydawnictwo IFiS PAN, Warszawa. 84. Thistlewaite D.L., Campbell D.T. [1960], Regression-discontinuity Analysis: An Alternative to the Ex post Facto Experiment, Journal of Educational Psychology, n. 51. 85. Trochim W. [1984], Research Design for Program Evaluation: The Regression-discontinuity Approach, Beverly Hills, CA, Sage Publications. 86. Trzciński R. [2009], Wykorzystanie techniki propensity score matching w badaniach ewaluacyjnych, PARP, Warszawa. 87. Rules of Procedures of the Council of Ministers as of 29th October 2013. 88. Weiss, C.H. [1995]. Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families, in: Connell J., Kubisch A.C., Schorr L.B., Weiss C.H. [ed.], New approaches to evaluating community initiatives: Volume 1, concepts, methods, and contexts, New York, NY: Aspen Institute. 89. Weremiuk A. [2012], Mierzenie efektów interwencji publicznych – opracowanie na potrzeby przygotowania poradnika ewaluacji, Warszawa. 90. White H. [2009], Theory-Based Impact Evaluation: Principles And Practice, Working Paper n. 3, International Initiative for Impact Evaluation, New Delhi. Internet sites: 1. http://data.worldbank.org/ 2. http://edukacjamedialna.edu.pl/lekcje/sztuka-dyskusji/ 3. http://www.ewaluacja.gov.pl 4. http://www.mg.gov.pl/Wspieranie+przedsiebiorczosci/Wsparcie+finansowe+i+inwestycje/Specjalne+strefy+ekonomiczne/Efekty+SSE 5. http://ideamappingsuccess.com/ 6. intranet.gov.pl 7. http://legislacja.gov.pl 8. http://www.mir.gov.pl, 9. https://www.oecd.int/olis/ 10. http://www.stat.gov.pl

139

Information about authors STANISŁAW BIENIAS Since 2004, Stanislaw Bienias has been working in the Polish Ministry of Regional Development in the National Evaluation Unit. He became Head of this unit in 2006. He was responsible for organizing the process of evaluation for the National Development Plan 2004-2006 and the National Strategic Reference Framework 2007-2013 and negotiation of regulations for 2014-2020 (evaluation, monitoring&indicators). Stanislaw was responsible for coordination of more than 50 evaluation units located on OP and intermediate body level. He took part in preparation of more than 100 evaluation researches on regional, national and international level. In the field of evaluation Stanislaw successfully cooperated with OECD, World Bank and UNDP. He was organizer and speaker on dozens of national and international conferences. Stanislaw was representing Poland on the DG Regio Evaluation Network and was one of the Polish experts in ESF Evaluation Partnership Meetings. He was also one of the animators of Visegrad Countries cooperation in the field of evaluation and analysis. He gives lectures concerning evaluation, monitoring and indicators in Warsaw School of Economics, Warsaw University and European Academy in Berlin. He is an author of numerous publications in the field of evaluation and public policies. Since 2013 he has decided to continue his mission within non-for-profit organization Foundation IDEA for Development.

KATARZYNA HERMANN-PAWŁOWSKA Katarzyna Hermann- Pawlowska graduated from the Faculty of Law and Administration at Adam Mickiewicz University in Poznań, from Interdisciplinary Postgraduate Studies at College of Europe in Natolin and from PhD studies at Warsaw School of Economics. In 2013 she obtained PhD in economics with specialization in public administration and European integration. She has a long term experience of working in public administration at regional level (Wielkopolska Voivodeship Office) and central level (Ministry of Re-

140

gional Development). She took part in preparation of the management system of the Cohesion Policy in Poland for 2007-2013 as well as was responsible for the institutional capacity building for managing and implementing EU funds in this period. Since 2008 she has been working as an independent expert and cooperating with public administration in the area of good governance, strategic management and institutional development. Author of publications concerning processes of Europeanization of Polish public administration, multi-level management and strategic planning. Member of Regional Studies Association since 2008. She gives lectures concerning EU Cohesion Policy and is a lecturer at Warsaw School of Economics. Since 2013 Katarzyna Hermann-Pawłowska is Strategic Project Director in IDEA for Development Foundation. SABINA KASOŃ Graduate of the International Relations at the Katowice University of Economics and the Programme of European Studies of the Warsaw School of Economics and SciencePo in Paris. For 12 years she has worked with the government administration in the Ministry of Infrastructure, Ministry of Regional Development and Ministry of Administration and Digital Affairs. Manager of many training and international cooperation projects financed by national and international funds. Expert in strategic planning. Co-author of National Development Strategy 2020: Active society, competitive economy, efficient state, Policy paper on digital development of Poland till 2020 and Operational Programme Digital Poland project. Since September 2014 Managing Director in IDEA for Development Foundation MACIEJ KOLCZYŃSKI Graduate of Faculty of Law and Administration at the Cardinal Stefan Wyszynski University in Warsaw. He gained his professional experience while working for institutions responsible for planning and management of EU Structural Funds as well as regional policy on national level, ia. the Ministry of Economy and Labour, Ministry of Regional Development, Cabinet of the Commissioner for

141

Regional Policy Danuta Hübner and in the Ministry of Infrastructure and Development. He was involved in the preparation of strategic and programming documents regarding the use of Structural Funds in Poland in 2007-2013 and 2014-2020 period. Member of the team elaborating the National Strategy for Regional Development, engaged in the preparation of the implementation tools for this strategic document. Speaker during international conferences and national seminars on regional policy. He was often engaged as an expert in international cooperation projects, also within the Polish Aid. He is also an author of articles in the field of territorial dimension of Cohesion Policy. Currently he is a key expert of the Foundation Idea for Development in the field of: regional policy, territorial self-government, urban development, international cooperation, institutional capacity.

KATARZYNA LOTKO-CZECH Ph.D. graduated from the Institute of Economics at the University of Opole. She is a University of Opole and Evaluation of Social Programmes at the University of Warsaw graduate. She worked in the Marshal’s Office of the Opolskie Voivodeship in 19992014. She built from the ground up the Evaluation Unit in the Department of the Regional and Spatial Policy of Marshal’s Office in Opolskie in 2007. She also prepared and was coordinating the process of the evaluation of operational programmes managed and implemented at the regional level. She was Vice-Chairman of the Evaluation Steering Group in Opolskie voivodeshp. She was also participating in works of the Evaluation Steering Team for Cohesion Policy and the Evaluation Steering Group for Human Capital Operational Programme in Ministry of Infrastructure and Development. She participated in preparing the main regional documents: Regional Development Strategy for the Opolskie Voivodeship until 2020, Regional Operational Programme of Opolskie 2007-2013, 2014-2020. She is a member of Polish Economic Society. An author and co-author of scientific articles, among others about: regional development, regional policy, European Union Cohesion Policy, public finance. The lecturer of many conferences and scientific seminars concerning regional policy and evaluation culture building on the regional level.

142

She gives a lecture, concerning mainly regional policy, in Higher School of Banking in Wrocław- Institute of Economics in Opole. The founder and member of the management board of the firm CG2 Sp. z o.o. [Ltd.]. RAFAŁ TRZCIŃSKI Rafal Trzciński – analyst, a graduate of the Institute of Sociology at the University of Warsaw. From 2005 to 2014, he worked in the Evaluation Unit of the Polish Agency for Enterprise Development. From 2014 he collaborates with the IDEA Foundation and I-SEE Foundation (Institute for Socio-Economic Enquiry). Since 2014 he also works in the Institute for Educational Research. He specializes in social research methods, data analysis, evaluation and analysis of public policies. Much of his work is devoted to the estimation of impact of various public programmes with the use of quasi-experimental approach. Author and co-editor of publications in this area. Member of the Polish Evaluation Society. ALICJA WEREMIUK Alicja Weremiuk is an experienced expert on public policy, evaluation the European Union’s development funds, including both pre-accession instruments and the EU’s cohesion policy. Mrs Weremiuk graduated from Warsaw University and obtained a diploma in International Relations. Currently she is a Ph.D student at Polish Academy of Science, Institute of Economics. Having worked for 5 years in Polish Ministry of Regional Development, she headed the evaluation unit for ISPA/Cohesion Fund (CF) and Infrastructure and Environment Operational Programme 2007-2013 (I&E OP). She was responsible for setting the CF evaluation system and evaluation plans, evaluation designs, ToRs elaboration, development of evaluation methods and tools, trainings on evaluation for co-workers. She also took part in I&E OP programming and negotiations. From 2008 Mrs Weremiuk worked as an independent evaluator and consultant. Her responsibilities vary from preparing tender bids, evaluation projects’ management, methodological co-ordination, conducting qualitative and quantitative researches, data analysis to report and recommendation writing.

143

During whole her career she also conducted a series of trainings, workshops and lectures on evaluation, research methodology, EUâ&#x20AC;&#x2122;s funds and public policy. Her consultancy experience covers projects in Poland, Georgia and Romania.

144

145

146

147

It is the quality of legal acts that influences the competitiveness of enterprises, the security of economic trade, the security of consumers and their relationships with business as well as the functioning of state in the social , economic and administrative dimensions. Developed countries have been widely taking advantage of the impact assessment system with regard to legal acts. In Poland initiatives in this area have been already undertaken for many years. The Handbook developed by experts of the IDEA for Development Foundation entitled ‘The Regulatory Impact Assessment. A Guide for Public Administration’ makes a great contribution to improving the impact assessment process with regard to legal regulations. The authors of this publication, well-experienced evaluators of many public interventions, including EU funds, have successfully faced an extremely difficult task which is transferring evaluation experiences into regulatory impact assessment. This Guide points out how social research can be effectively and efficiently used for improving the quality of legislation in Poland. The value added of the Handbook is the use of differential methodological approaches such as evaluation based both on theory and counterfactual methods. All the suggested initiatives to be undertaken in the impact assessment process have been illustrated by practical examples which enable the Reader to better understand subsequent research steps and their consequences for the ultimate results. Hence, the Handbook is a significant tool for officials of public administration who prepare the impact assessment of draft legal acts both before and after their implementation. Prof. Adam A. Ambroziak Ph. D. J. Monnet Chair of European Integration Collegium of World Economy, Warsaw School of Economics The publication content has been to great extent inspired by the evaluation system of public interventions, the history of which has been one decade longer in Poland as compared to the regulatory assessment system. The Handbook accurately depicts similarities of the two systems which justify their mutual benefiting from the best standards. I regard such an approach as a great strength of the Handbook (...). The publication fits in well to a growing need of analysts, administration practitioners, decision-makers as well as other stakeholders for conducting reliable and credible regulatory impact assessment. (...) The Handbook provokes thoughts and further research aiming at professionalization of decision-making and public initiatives. It accounts for an invaluable source of information and great reading for all the interested in improving the regulatory impact assessment process. (...) Hence, publicizing the Handbook will contribute to increasing the analytical capacity of Polish administration. Dr Marcin Sakowicz National School of Public Administration The Handbook ‘The Regulatory Impact Assessment. A Guide for Public Administration’ developed by experts of the IDEA for Development Foundation is a most interesting publication which may prove to be extremely useful for officials of public administration dealing with research on the impact of legal acts. This subject publication clearly and comprehensibly synthesizes and describes all works necessary for conducting reliable research for assessing the regulatory impact ex ante, ex post or as for regulatory test (…) The Compendium prepared by the experts of the IDEA for Development Foundation (…) is based on experiences of the Polish system of Cohesion Policy evaluation. These experiences are relevantly adapted to the needs and challenges facing evaluators of a new research area concerning a mode of creation and operation of legal acts in Poland. Thanks to it, the publication is complementary to the existent developments, which significantly influences its utility. Piotr Strzęboszewski National Evaluation Unit Ministry of Infrastructure and Development