AJE281701

American Journal of Evaluation http://aje.sagepub.com

Toward Theory-Led Evaluation: The Experience of European Science, Technology, and Innovation Policies Jordi Molas-Gallart, Jordi Molas-Gallart, Andrew Davies and Andrew Davies American Journal of Evaluation 2006; 27; 64 DOI: 10.1177/1098214005281701 The online version of this article can be found at: http://aje.sagepub.com/cgi/content/abstract/27/1/64

Published by: http://www.sagepublications.com

On behalf of: American Evaluation Association

Additional services and information for American Journal of Evaluation can be found at: Email Alerts: http://aje.sagepub.com/cgi/alerts Subscriptions: http://aje.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations (this article cites 19 articles hosted on the SAGE Journals Online and HighWire Press platforms): http://aje.sagepub.com/cgi/content/refs/27/1/64

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

American Journal 10.1177/1098214005281701 Molas-Gallart, Davies of Evaluation / T oward /Theory-Led March 2006Evaluation

Toward Theory-Led Evaluation The Experience of European Science, Technology, and Innovation Policies Jordi Molas-Gallart INGENIO (CSIC-UPV)

Andrew Davies Tanaka Business School, Imperial College London Abstract: This article reviews the literature and practice concerned with the evaluation of science, technology, and innovation (STI) policies and the way these relate to theories of the innovation process. Referring to the experience of the European Union (EU), the authors review the attempts to ensure that the STI policy theory is informed by advances in the authors’understanding of the innovation process. They argue, however, that the practice of policy evaluation lags behind advances in innovation theory. Despite the efforts to promote theory-led evaluations of STI policies based on new theories of the systemic nature of innovation, evaluation practice in the EU continues to favor the development of methods implicitly based on outdated linear views of the innovation process.

This article examines the reasons why this is the case and suggests that STI policy evaluation should nevertheless be supported by the evolving theoretical understanding of the innovation process.

Keywords: policy evaluation; science, technology, and innovation policy; innovation theory; Framework Programme evaluations

T

he evaluation of science, technology, and innovation (STI) policies is challenging because of the multiple goals of STI policy, the indirect and complex linkages between policy outputs and outcomes, and the serendipitous and long-term nature of policy impacts (Feller, 2002; Molas-Gallart, Salter, Patel, Scott, & Duran, 2002). This article reviews the literature and practice concerned with the evaluation of STI policies, focusing on the evaluation of European Union (EU) programs. During the past two decades, the evaluation of STI policies in the EU has been undertaken by a community of specialist academics and niche consultancies. Although some of the problems faced are common across different fields of evaluation, this community and the practices of STI policy evaluation have had limited exposure to other evaluation communities.

Jordi Molas-Gallart, Research Professor, INGENIO (CSIC-UPV), Camí de Vera s/n, Universitat Politècnica de València, 46027 València, Spain; e-mail: jormoga@ingenio.upv.es.

Authors’Note: The research that led to this article was conducted while both authors worked at SPRU, University of Sussex, as senior fellows. This article further develops arguments that the authors presented in a report commissioned by the Swedish Institute for Growth Policy Studies (Institutet for Tillväxpolitiska Studier).

American Journal of Evaluation, Vol. 27 No. 1, March 2006 64-82 DOI: 10.1177/1098214005281701 © 2006 American Evaluation Association

64

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 © 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 65

In this article, we focus our attention on the ex-post evaluation of the results of policy initiatives and not on the ex-ante assessment of project proposals submitted to an agency to obtain funding. The typical policy instrument is that of a program by which a public organization funds a set of research, technological development, or other innovation-oriented activities to be carried out by firms, research organizations, or both. Funding is commonly in the form of subsidies, covering up to 50% of the project’s cost in the case of private beneficiaries or up to 100% in the case of public research organizations (e.g., higher education institutions). External, independent experts usually undertake the evaluation of these programs once they have been completed or are at an advanced stage and likely to have produced some results. Oftentimes, the agencies managing the programs do not have the breadth of capacities and expertise necessary to assess the results of a program and will require an independent view on the results of a specific initiative. For this, they rely on external consultants engaged after open or restricted requests for proposals. A community of specialist STI policy evaluators has developed to supply this need. Small specialized consultancies share this evaluation market with individual experts and groups from larger research organizations for whom policy evaluation is one among a broader portfolio of activities usually linked to the analysis of innovation and public policy.

The role of the STI policy evaluators is to determine what the policy outputs and outcomes have been, in other words, to link changes in the behavior of project participants and the impacts that such changes have generated to the specific policy initiatives under evaluation. As we will see below, policy makers are commonly interested in the impacts of STI policies on, for instance, productivity, competitiveness, and social welfare.

In the field of STI policy, both policy and program theories are strongly rooted in innovation theory. By policy theory, we understand a broad set of assumptions underlying generic policy goals and approaches (e.g., the belief that the support of basic research will promote academic excellence, economic competitiveness, and welfare in general). In contrast, following the conceptual distinction made among others by Leeuw (2003, p. 6), program theory refers to the rationale underpinning a particular policy initiative (a specific policy program), specifying its inputs and components, its expected outcomes, the assumed linkages between outcomes and policy inputs, and the underlying mechanisms that are responsible for these linkages. As the article will show, in our area, new policy approaches can be traced to changes in innovation theory. We will discuss how our understanding of the innovation process and the role that research and development (R&D) plays within it has influenced STI policies and how evaluation practice has tried to adapt to the changes. The role of the evaluator is to develop further the logical links between policy practice and their expected effects and to turn them into the theoretical support for a detailed evaluation study. In other words, the evaluator needs to develop a specific, detailed program theory from the policy theory that underpins the initiative under evaluation.

This article argues, however, that unlike STI policy theory, the practice of policy evaluation continues to lag behind advances in innovation theory. Innovation theory has produced successive generations of more sophisticated conceptual models that seek to explain how the relationship between scientific and technological research and the market opportunities for innovation occurs.

In this article, the term innovation refers to the entire set of processes linking R&D to the market introduction of a new product or service. Initial views postulating a linear relationship between scientific research, technological development, and eventually innovation and competitiveness informed the program logic of early STI policies. Currently, the new theories on the systemic nature of technological innovation provide a new STI policy theory, emphasizing the importance of the relationships between different actors in the innovation system and the com-

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

66 American Journal of Evaluation / March 2006

plex connections between R&D activities and the whole of the innovation process in which they are inserted. The evaluation community has reacted by proposing new ways to adapt the evaluation of STI policies to the new policy theories. Yet such change is difficult to implement in practice because it involves a complete shift in the traditional approaches to STI policy evaluation and, therefore, in the way in which policy agencies contract for and use external evaluations. We can describe this situation as a lag between evaluation practice and STI policy theory.

This lag between evaluation practice and policy theory is not confined to the STI policy evaluation arena. Evaluation in many other fields needs to assess, and sometimes is required to measure, the more elusive downstream, societal impacts of policy programs. Our article is concerned with instances where evaluators are asked to appraise the impacts of policies whose effects are likely to emerge in the medium and long term and that are difficult to trace and attribute to the original policy measures. We are speaking here of complex environments where policy measures are only one among many other factors with a bearing on those variables that the policy community is trying to influence. In the case we are addressing, STI policy, complex theories of change have been developed to account for the intricate way in which different activities and factors can explain the innovation process. We discuss how these theories have proved difficult to use in the practice of evaluation, resulting in a gap between evaluation practice and STI policy theory. This gap is a result of the tension between complex theories of change and the simpler models required to deliver accountability-focused performance measurements.

The problem is relevant because in almost all areas of public policy, the trend toward resultsbased management in public administration has had the effect of focusing attention on the measurement of policy outcomes (rather than outputs). Such outcomes commonly take place downstream from policy action as policy outputs combine with other factors. In this context, observable changes in target areas are difficult to attribute to specific policy actions; in response evaluation, practitioners have proposed multifaceted approaches to the analysis of the contribution of policies to specific outcomes (Mayne, 2001). Similarly, as we will see, STI policy evaluation experts have also suggested multilevel evaluations, trying to capture the current understanding of the complex and systemic nature of innovation processes, in which STI policy is inserted. We will argue, however, that the assessments that these evaluations will yield will not satisfy the politically driven requirements for simple quantitative measures of program performance: mainly, the increasingly repeated request that evaluation be able to deliver a single figure demonstrating the economic returns derived from specific policy initiatives.

The article first sets the scene for the analysis of innovation policy evaluation by considering how models of innovation have produced insights about the functioning of innovation processes. These models have important implications for policy: They provide a rationale (or policy theory) for government intervention in support of innovation. The shift in the rationale for policy intervention should drive changes in evaluation theory and practice. In the second part of the article, we discuss recently proposed changes in the focus of STI policy evaluation. Evaluation experts have proposed new approaches based on a systems-based understanding of innovation. In general, they suggest a shift away from the evaluation of the direct outcomes of individual programs or projects toward attempts to assess the effects of policy on the performance of innovation systems as a whole. Yet using examples from the evaluation of EU STI policies, the section will show how other political trends are driving evaluation policy in a different direction. We identify a tension between political demands for accountability and outcome measurement and the complex results yielded by systemic approaches to evaluation. The article concludes that the role of the evaluator is central in steering the evaluation practices toward the implementation of approaches informed by our current understanding of innovation processes.

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 67

Evolving Models of Innovation and the Rationale for Policy Intervention Since the 1950s, numerous empirical studies have been conducted to develop a theory of innovation. These studies have produced successive generations of conceptual models based on increasingly sophisticated explanations of the process of innovation (von Hippel, 1988). Each model has furnished new empirical evidence pointing out the limitations of the previous generation and, in so doing, questioned the assumptions of previous approaches to innovation policy.

Policy makers seek to understand the factors that influence the sources, rate, and direction of innovation so that they can formulate policies to achieve economic and social benefits. A link can be found between the trends in STI policies and our theoretical understanding of the relationships between R&D, innovation, and wealth generation. This section reviews the main theoretical advances in the study of innovation and the changes in policy practice that followed. We show how the evolving theoretical understanding of the relationships between R&D, innovation, and wealth generation are linked to changes in policy approaches.

Linear Models of Innovation: Supply Push and Demand Pull In the decades after World War II, science policy was based on the belief that scientific research would result in beneficial social and economic outcomes. This belief was laid out in the influential 1945 report Science the Endless Frontier (Bush, 1945). The report to the U.S. president, authored by Vannevar Bush, director of the Office of Scientific Research and Development, argued that scientific research was key to U.S. welfare and security and that essential new knowledge could only be obtained through basic scientific research. Government support to basic research and the development of scientific talent would translate into new products and technology, leading to social and economic improvements and enhanced military security. In other words, basic research forms the basis for new knowledge on which, in turn, applied research and technological development rest for the development of new products and processes. This view of the role of R&D provided the rationale underpinning U.S. and European research policies in the decades after World War II. Such supply-push models of innovation were linear in that they saw the generation of new knowledge and technology as following sequential stages from basic research through applied research to technological development and exploitation in the market. The supply-push model—also known as “science push” and “technology push”—emphasized the role of original research and invention and downplayed or ignored market demand. The policy implications of the supply-push model are that spending money on R&D promotes innovation and stimulates economic growth. Large government expenditures on research are justified on the grounds that they expand the opportunities for innovation. The policy outcome is supply-side intervention providing public funds for basic, undirected research and applied research. These policies were popular in the 1950s and 1960s, when there was a great deal of optimism about the potential of research to meet social and economic objectives.

From the mid-1960s to the late 1970s, empirical studies criticized supply-push approaches for downplaying or ignoring the role of market forces in the innovation process. These studies argued that innovations are triggered in response to consumer and user demands (Schmookler, 1966). Such demand-pull models of innovation were also linear but stressed the role of the market in pulling technological innovation and directing R&D investments.

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 © 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

68 American Journal of Evaluation / March 2006

Nonlinear Models of Innovation Coupling Models (Late 1970s to 1980s) By the late 1970s, studies of innovation concluded that there was little evidence to support the view that demand-pull forces govern the innovation process. Furthermore, it was argued that demand-pull theories provide little help in formulating policies aimed at promoting R&D and innovation (Mowery & Rosenberg, 1979, p. 194). Several authors (Freeman & Soete, 1997; Kline, 1985; Kline & Rosenberg, 1986; Rothwell, 1992) saw innovation as a two-sided process involving complex interactions between supply and demand: • The supply side includes new scientific or technical knowledge that is the result of original research activity conducted by various science and technology institutions (R&D laboratories and scientific and technical institutions). • The demand side is derived from actual or perceived market demands for new technologies, products, services, or processes.

The efforts to develop a more comprehensive theory of innovation aim to account for the interaction, or coupling, of both elements (Freeman & Soete, 1997, p. 201). As depicted in Figure 1, the coupling model of innovation addresses the direct and indirect results of the innovation process arising from the continually changing interfaces between advancing science, technology, and the market (Rothwell, 1993). Unlike the previous linear models, the coupling model links decision making within the firm to basic research and to the marketplace. Coupling models treat innovation as a nonlinear, chain-linked process involving communication paths between functionally interacting and interdependent stages, with feedback loops between later and earlier stages (Kline, 1985; Kline & Rosenberg, 1986).

Innovation Systems Models (Early 1990s to Early 2000s) During the 1990s, the coupling model of innovation was refined to take into account the more complex, distributed, and often random nature of the innovation process. New models were developed to account for the network of interactions linking science, technology, and the marketplace, as well as the risks and uncertainties associated with policy making. In the early 1990s, attempts to develop a conceptual model were conducted from two perspectives: networks of firms and national systems of innovation. First, in an attempt to refine the coupling model of innovation, Rothwell (1992) recognized that innovation occurs increasingly in networks and alliances of firms rather than in the individual company or R&D laboratory. The increasing number of corporate alliances, partnerships, R&D consortia, and joint ventures pointed to the importance of cooperation between systems integrators and their suppliers, customers, and collaborating competitors. By the 1990s, a new pattern of innovation was emerging based on a distributed learning process taking place within and between networked firms (Rothwell, 1992; von Hippel, 1988).

Second, in contrast with this emphasis on the firm, a growing body of literature emphasized the role of the national or regional environments, including local institutions, organizations, and culture, in shaping the innovation process (Lundvall, 1992). These national or regional systems-of-innovation approaches are concerned with the ways in which institutional setups influence the rate and direction of innovation and focus on the interactions and learning

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 69

Figure 1 The Coupling Model of Innovation New need

Idea generation

Needs of society & the marketplace

Research, design & development

New technology

Prototype production

Manufacturing

Marketing & sales

Marketplace

State of the art in technology & production

Source: Rothwell, 1993, p. 21.

processes between the institutional actors involved. Because, for instance, users of knowledge are also knowledge producers, traditional distinctions between supply and demand break down.

These models have been developed to take into account nonlinear processes and feedback loops between different stages in the innovation system, underlining the complex links between R&D and innovation (Grupp, 2000). These nonlinear, nonsequential models of innovation can take into account problems arising at late stages of innovation, which can trigger basic research activities. In other words, research is an adjunct rather than a precondition for innovation and can relate to any stage in the innovation process.

More complex models of the innovation process are now adopting an innovation systems approach (Guy & Nauwelaars, 2003), informed by general systems theory, to capture the increasing complexity of innovation processes. Innovation systems comprise many actors (firms, governments, universities, public research institutes, users, etc.) and the relationships between them, underpinned by flows of information, finance, and power. Innovation systems theory calls for new systemic approaches to policy formulation, implementation, and evaluation. The next section discusses the way in which STI policy has adapted to this evolving understanding of innovation and the implications of this new policy rationale for evaluation practice.

Innovation Theories and Changing Approaches to STI Policy As a broad policy theory, supply-push models supported the belief that basic research would yield public goods; there was no perceived need for the establishment of concrete, more specific missions for policy programs. Soon, however, the need to justify such investments in terms of identifiable returns became a policy concern. This concern emerges as demand-pull models start to emerge. Within a demand-pull framework, R&D needs to fit with the needs of its users and beneficiaries; it cannot be taken for granted that basic research will automatically lead to societal benefits. Impact assessment tools were developed from the late 1960s to demonstrate the downstream benefits of publicly funded R&D. This is a difficult task as, even within a linear model, the impact of R&D is often long term and difficult to identify and trace. In the late 1960s

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

70 American Journal of Evaluation / March 2006

and early 1970s, retrospective studies were launched in the United States to assess the dependence of innovations on scientific work. The main examples of such retrospective approaches were Project Hindsight and Project TRACES, the former sponsored by the U.S. Department of Defense and the latter funded by the National Science Foundation (see, e.g., Kostoff, 1997; Sherwin & Isenson, 1967).

Demand-pull models influenced new attitudes toward government policies and funding. If innovation emerged as a consequence of specific demands, supply-side policies to accelerate technological progress through funding of basic research could be misguided. A new social contract was needed to steer scientific activities toward areas in which they could contribute to addressing societal needs. Scientific research started to be seen as a long-term investment responding to societal needs, and therefore, scientific priorities needed to be defined in a strategic manner.

This social contract between science and society required that the scientific community showed that its activities would address issues of economic and social relevance. The policy response was a new approach that focuses on strategic, targeted, and problem-oriented research.

The need to link basic and fundamental research with societal needs also led to an increased emphasis on the role of universities as generators of growth and on university-industry relations. It was argued that the traditional way of handing off basic research in a sequential manner from universities and government laboratories into industry “simply does not work quickly or effectively because of divergent focus, and lack of proper incentives among several other problems� (Betz, 1994, p. 786). In response, initiatives like the U.S. National Science Foundation Industry/University Cooperative Research Projects Program, developed between the late 1970s and the early 1980s, allowed university and industrialists to jointly identify new fundamental disciplinary approaches (Betz, 1994).

As an even more complex and systemic view of the role of R&D developed, policy continued to adapt accordingly. The linear approaches to STI policy do not sit well with the current understanding of the innovation process, with its feedback loops, interrelations, and inherent complexity. Not only is it no longer implicitly expected that any investment in basic research will naturally result in improved welfare, but the policy process itself should become more fluid and flexible to adapt to the complexity of the system of which it is part. During the 1980s and 1990s, different approaches, such as technology foresight,1 technology assessment,2 and formative approaches, to evaluation were being developed and used to aid STI policy formulation in a complex and distributed innovation system (Kuhlmann, 1999). STI policy formation in the EU and in many European countries has adopted several of these elements. Foresight exercises are, for instance, routinely conducted in several European countries, and the European Commission (EC) officials follow these exercises closely. The formation of an EU STI policy is also based on a broadly based consultation, informed by the results of systematic evaluation exercises. Furthermore, many of its policy initiatives address issues such as the promotion of networks and international training that are important in any systemic understanding of the innovation process. Yet Kuhlmann (1999, p. 19) argues that by the late 1990s, there was still no blueprint for combining all these tools in a single strategic intelligence approach integrating the different stages of the policy process (formulation, agenda setting, decision, implementation, and evaluation) in an interactive manner. For Kuhlmann, what is needed is for STI policy to be defined through portfolios of actions informed by interactive systems of evaluation constantly feeding into the policy formulation process. In other words, for Kuhlmann, developing a formative approach to evaluation becomes a key element in building up a systems approach to policy formation.

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 71

How Evaluation Practice Responded to Changing STI Policy Theories When compared with other policy fields, STI policy evaluation practice evolved relatively late. Recent reviews trace evaluation studies in the United States back to the late 1960s (Roessner, 2002) and in Europe to the 1970s (Luukkonen, 2002), although ex-ante assessment of R&D proposals, characterized mainly by peer review procedures, started much earlier (Rip, 2003). Noticeably, the concern to develop a systematic approach to policy evaluation developed relatively late when compared with other fields. By the late 1960s, evaluation practices and methods were already well established in fields such as the evaluation of education (Campbell & Julian, 1963) and social policies, and evaluation methodologies had already received substantial attention (Scriven, 1967). In contrast, in the early 1980s, the practice of STI policy evaluation was still in its infancy (Gibbons & Georghiou, 1987). There were few systematic guidelines as to how to evaluate major programs and few systematic evaluations of the effectiveness of innovation policies. Yet today, STI policy evaluation has become one of the most important areas of concern for STI policy makers. This interest can be traced to two interrelated factors: (a) the development of a new social contract between science and society discussed above and (b) the emergence of new public management approaches emphasizing the application to all public functions (including the funding of R&D activities) of management practices oriented to the control of outputs rather than simply monitoring processes and inputs.

Although the new social contract underpinning public funding of R&D activities is linked to a more complex view of the relationship between science and social and economic welfare, we argue here that developments in the management of public policies have pushed evaluation practice in another direction. The ascendancy of the so-called new public management approaches in the 1980s increased the pressure for evaluation to provide evidence of policy results and to develop output, rather than input, measures. It proposed the application of performance indicators, output controls, management by accounting, and increased competition and marketization of public sector activities (Boden, Gummett, Cox, & Barker, 1998). Increasing attention was paid to evaluations to assess and measure the performance of large R&D and innovation support programs. These evaluations focus on the development of measurement methods and techniques that do not refer to the more complex evolving STI policy theory. In the following sections, we will review the way in which evaluation experts are attempting to incorporate current policy theories into evaluation practice and how their proposals may enter into conflict with the political need for performance measurement. Theory-led evaluation may clash with the pragmatist search for new measurement methods and result in a conflict between method and theory-led evaluation that has already been noted in other policy fields (Stame, 2004).

Changing Approaches to the Evaluation of STI Policy Attempts at evaluating STI policy initiatives became widespread in the 1980s, when STI policy theory was driven by demand-pull theories of innovation. The concern about the use of research results and the alignment of STI programs with societal needs resulted in a new social contract between science and society, which in turn had implications for the evaluation of science and technology policies (see, e.g., Dalpe & Anderson, 1993; Jaffe, 1998). Strategically targeted R&D called for evaluation practices able to assess whether the specific policy goals had been achieved and thus support the decision-making process.

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

72 American Journal of Evaluation / March 2006

By the mid-1990s, analysts of STI evaluation were noting that the assessment of the linkages between the scientific community and potential users and beneficiaries had become a dominant element of STI initiatives and their evaluation (Dietz, 2003; Luukkonen, 1998). Yet at the same time, it was starting to become apparent that a monitoring and evaluation approach in line with the developing systemic approaches to policy development (i.e., policy initiatives rooted in a systemic policy theory) would be more complex. Different evaluation approaches are being proposed.

Formative Evaluation As discussed above, Kuhlmann (1999) has aligned the implementation of a formative approach to STI policy evaluation with the development of a systemic approach to the policy formation process. In this view, different policy initiatives would no longer be considered and planned in isolation from each other but rather as part of a wider portfolio of actions addressing the performance of the system at different levels. STI policy would then comprise interlinked portfolios of initiatives, which would be informed by interactive systems of evaluation, constantly supporting the policy formulation and implementation processes. Such systemic approach to policy formation stems from the heightened awareness of the complex relationships between knowledge creation and innovation, and the diversity of actors involved in these processes, and rests on a formative approach to evaluation, which becomes a key integral part of this policy model. On their own, formative approaches are not new in the field of STI policy evaluation. For instance, the evaluation of the U.K. Alvey program (1984 to 1990), a British initiative to support R&D in the information technology sector, used new combinations of evaluation techniques to develop a real-time evaluation approach and can be seen as an early example of formative STI policy evaluation. The Alvey evaluators argued that real-time evaluation had several advantages over traditional ex-post evaluation, significantly among them the ability to feed back evaluation results to those responsible for directing the program (Hobday, 1988).

Kuhlmann goes further and proposes a move away from an objective model of evaluation, in which independent evaluators produce evidence but no recommendations, and toward a model involving evaluators in learning exercises with all stakeholders and providing advice and recommendations as well as independent analysis. In this formative context, the evaluator becomes a facilitator rather than an external expert. The result would be a more flexible and experimental approach to policy formulation.

The move to formative evaluation has important methodological implications (Guy, 2003; Kuhlmann, 2003). Beyond the need to adopt real-time evaluation and pay less attention to impact assessment, which by their own nature requires a delayed time frame to be carried out, formative evaluation needs to engage the stakeholders in charge of policy development and implementation into the evaluation process. Proponents of formative evaluation in the STI field argue that ex-post, hands-off evaluation by external experts is no longer an option. Evaluation has to become a process, part of policy implementation, by which programs are constantly assessed to improve the policy process.

Systemic Analysis Arnold (2004) suggests a multilevel system of STI policy evaluation, encompassing traditional project and program evaluation, the systemic analysis of the overall health of innovation systems, and a subsystems analysis of bottlenecks, exploring the role of institutions and other actors. This systemic approach to evaluation would address institutional conditions, the con-

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 73

nectivity of the system (cooperation and networking across institutional boundaries), and the system capabilities, from a knowledge, economic, and technical point of view (Arnold, 2004).

Other experts are also placing greater emphasis on the evaluation of STI policy portfolios (Guy, 2003) and the linkages between evaluation and complex processes of policy formation.

Intelligent Benchmarking A different response to cope with the new approaches to STI policy and their underlying systemic policy theories is the attempt to assess the effect of policies on the innovation system using so-called intelligent benchmarking approaches (Guy & Nauwelaars, 2003; O’Doherty & Arnold, 2003; Soete & Corpakis, 2003). Benchmarks aim to establish standards to measure and assess performance and to show where improvements are possible. They offer the promise of providing sets of measurements that may be easy to communicate while rooting evaluation practice on the current systemic understanding of the nature of innovation and technological change. Benchmarking proponents argue that it can be used to assess whether STI policies are having the expected downstream benefits on competitiveness and employment.

The EC is supporting the development of intelligent benchmarking. For instance, it has funded a high-level working group to study the benchmarking of national STI policies in relation to the Impact of RTD on Competitiveness and Employment (STRATA-ETAN Expert Working Group, 2002). This working group developed benchmarking frameworks for the performance (in terms of improved competitiveness and employment) of innovation systems and for the policies likely to affect this performance. The approach was informed by concepts furnished by the innovation systems literature. It defined four categories (social and human capital, research capacity, technological and innovation performance, and absorptive capacity) under which to group sets of indicators to help define the performance of an innovation system. The four categories can be related to each other by a model of an innovation system (Guy & Nauwelaars, 2003). The research concluded that “benchmarks of performance along these dimensions . . . are thus highly desirable as inputs to improved policy making” (STRATAETAN Expert Working Group, 2002, p. ix).

The Organization for Economic Co-operation and Development (2002) has also developed a similar benchmarking approach to examine industry-science relationships in national innovation systems. The approach includes a conceptual framework for assessing the relationship between industry and science based on three dimensions of a national system of innovation: (a) channels of interaction between organizations, such as companies, research institutes, and other bodies; (b) incentive structures; and (c) institutional arrangements.

The proponents of intelligent benchmarking are, however, aware of the practical difficulties of using benchmarking as a policy evaluation tool (Soete & Corpakis, 2003, p. 4). Benchmarking how policies affect the performance of innovation systems is complicated by three main obstacles: • The complexity of innovation systems—in terms of the number of activities and actors involved— makes it extremely difficult to identify the causal links between R&D on the input side of such systems and competitiveness and employment on the output side.

• Even if adequate models were available to draw correlations between input and output indicators, it is very difficult to isolate and assess the impact of policy initiatives on system performance. The scale and direction of R&D is primarily driven by private sector decision making rather than public policy initiatives.

• The problems of complexity, causality, and attribution conspire against the formulation of generic policy lessons and prescriptions. The multidimensional complexity of innovation systems suggests

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 © 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

74 American Journal of Evaluation / March 2006

that policy initiatives that work well in one context might not generate similar benefits in other contexts.

Analyzing the aggregate impact of multiple policy instruments is also difficult. Although it is possible to make crude correlations at the macrolevel between indicators such as government R&D expenditures and innovation system performance indicators, such quantitative calculations will tell us little about the links between policy and the observed impacts, the efficacy of particular policy mixes or individual instruments, or the specific policy levers that have to be pulled to improve overall system performance.

Therefore, according to its proponents, benchmarking cannot provide a simple, univocal measure of the impact of a specific policy or a group of policies. It can only provide data to populate a complex analytical framework where different performance factors and policy instruments are combined and lead to results across a range of variables. It will seldom be possible to attribute a change in measured performance to a specific policy measure. Therefore, benchmarking appears more as a further tool to aid in policy definition than an impact measurement approach to the evaluation of project and program performance. It must be noted, however, that in practice, the recent popularity of benchmarking does not adhere to this model.

Benchmarks are now being used for broad comparisons of national innovative performance, with targets being set up for a very small number of the most common indicators. As an extreme case, take for instance the EU targets for investment in R&D set up during the 2002 meeting of the European Council in Barcelona. The Barcelona targets set up the goal for all EU countries to spend 3% of their gross domestic product in R&D activities by 2010 (up from 1.9% in 2000) and to increase the level of business funding to two thirds of all R&D expenditure. This type of benchmarking has become very important in European STI policy but clearly falls far short from being part of the systemic approach to formative policy evaluation that its proponents support.

Evaluation in Practice: Implementing New Approaches to EU STI Policy Evaluation Changing STI policy theories have set a challenge for evaluation practice. As the results of R&D activities are inserted in a complex system and recombined with multiple other sources of innovation and are themselves affected by the results of previous R&D, impact assessments for R&D projects and programs become very difficult to carry out. As discussed above, European evaluation experts have proposed different approaches to meet the challenge. Intelligent benchmarking, systemic evaluation, and formative approaches to evaluation take as their point of departure the need to deal with a systems theory of innovation. They can therefore be described as theory-led evaluation approaches. Yet these approaches are unlikely to yield unequivocal measures of program impact.

The accountability culture that is developing in public management is driving politicians and public officials to develop clear, unambiguous performance indicators focusing on output and impact measurements. For instance, it is common among evaluation experts involved in evaluations of EC’s Research and Technological Development Framework Programmes (see below) to be asked to provide more specific measures of project and program impact. This has encouraged a focus on method-led approaches to evaluation and, in particular, the development of quantitative methodologies able to yield numerical assessment of outcomes and impacts.

This section discusses the way in which the tension between method-led and evolving theory-led evaluations is being resolved in the case of the EC’s long-standing efforts to assess consecutive Research and Technological Development Framework Programmes. The Frame-

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 75

work Programme constitutes the most important set of initiatives to fund EU research and technological development activities. It aims to strengthen the scientific and technological basis as well as the competitiveness of the European industry. The Programmes define the main research priorities and objectives structured in a series of Specific Programmes and establish a set of funding instruments and an associated budget to cover a period of 5 years. The 1st Framework Programme was launched in 1984, and in 2005, the 6th Framework Programme is underway, and discussions for the definition of the 7th Programme are already well advanced. The EC’s experience in the evaluation of the Framework Programmes is particularly relevant for the evolution of STI policy evaluation in Europe.3 A steady flow of STI evaluation work has developed around their assessment dating back two decades (Georghiou, 1995, 2003). For instance, between 1984 and 1994, more than 70 program evaluations and more than 40 supporting studies were conducted (Karatzas & Fayl, 1999). These evaluation activities have led to the consolidation of a sizeable European evaluation community.

Framework Programme evaluations have evolved toward increased institutionalization and emphasis on the contribution of external experts, performance assessment, and systems of continuous monitoring (Guy & Arnold, 1998). A 1990 report (Policy Research in Engineering Science and Technology [PREST], 1990) pointed out that in practice, the evaluation efforts had served mainly to reinforce and legitimize existing policies. The need to provide harder evidence of program performance led to the legislative requirement in the 4th Framework Programme to take a more systematic approach to evaluation and the reorganization of the process in 1994.

The new system set up an evaluation process, which is by and large still followed today, based on continuous monitoring conducted by program managers supplemented by assessments carried out annually by expert panels focusing mainly on implementation and 5-year assessments also conducted by expert panels. Oftentimes, the work of the panels is helped with evaluations of specific initiatives contracted out to expert evaluators. The 5-year assessment is particularly important: It is conducted halfway through the 5-year program and includes an ex-post evaluation of the previous program, plus a midterm review of the current one. In this way, the evaluation feeds on to the management of the ongoing program and contributes toward the definition of the following one. This approach attempts to bring together evaluation tools for real-time monitoring and the evaluation of project results; the former is to be used to introduce improvements in program implementation and the latter to inform the allocation of budgets and the definition of instruments and R&D priorities in subsequent programs. In practice, the approach results in a system of rolling overlapping evaluations conducted at different levels and feeding into the policy process also at different levels. Structuring a continuous evaluation process with policy definition represents an instance of systematic integration of policy definition with evaluation practice (see, for instance, Georghiou, Ribgy, & Cameron, 2002; Kastrinos, 1994; Laredo, 1998; Luukkonen, 1998; PREST et al., 2002).

Yet having a structured, systematic process in place does not solve the problem of the approach to be taken by the expert panels and the evaluators conducting specific evaluation studies. In particular, politicians have found the results of the process unsatisfactory. For instance, in 1997, the European Parliament was openly critical of the results of the annual evaluation report, stating that the report did not present data of an evaluative nature that would help them scrutinize the results of each of the Specific Programmes constituting the Framework Programme. European parliamentarians asked for data on, among other issues, the contribution of the initiative to the strengthening of the scientific and technological bases of industry and its competitiveness and the impact of the research on the labor market, quality of life, and the environment (Guy & Arnold, 1998).

This is an example of how the officials at the EC and, through them, the community of STI policy evaluators are being asked to provide measurements of the program’s economic and

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

76 American Journal of Evaluation / March 2006

social impact. It is not a unique case. During 2003 and 2004, one of the authors was involved in an evaluation of the types of projects funded under the 5th Framework in the specific field of information technologies (Hawkins, Montalvo-Corral, van Audenhove, & Molas-Gallart, 2004). The request for proposals specified a case study methodology to identify instances of good practice that could be used in the definition of actions and the fine tuning of management processes in forthcoming initiatives. In the initial phases of the study, we used cases to analyze the way in which research groups in international teams related with each other and how they interacted with project managers in different types of projects. Halfway through the project, however, our client started to require quantitative evidence of differentials in performance across instruments. The need for a simple measure to determine which types of actions were working well and which not so well or not at all became an objective of the project as it became clear that many of the examples of good practice we were finding were not contingent on the type of project or funding instrument. The need to show hard numerical evidence of performance was clearly important to the client in the process leading to the definition of R&D priorities in the next Framework Programme. This is a situation in which different research priorities and therefore groups within the EC working in different fields enter into political competition for control over the management and distribution of scarce funding resources.

Both of these examples show recent instances of situations in which program administrators find themselves under pressure to provide simple data reflecting program impact. The political pressure for output and impact measures can be linked to the diffusion and popularity of New Public Management approaches to public administration. New Public Management has emphasized the need to develop a culture of accountability in the public sector, revolving around the monitoring of policy achievements usually in relation to quantitative targets. The requests of the European Parliament and EC officials are for clear, easy-to-interpret data derived from impact assessments. Moving toward impact measurements poses a substantial challenge. It is the complexity of the innovation process and our understanding of it, as reflected in the systemic theories of innovation discussed above, that makes it difficult to reduce evaluation to a measurement exercise. Besides, the complexity of the relationship between R&D, innovation, and eventually economic or social changes means that the impact of a research initiative may not be felt for years after it has been completed. Identifying and tracing these effects becomes more difficult the longer the time elapsed since the project finished. (For a practical example of how these problems emerge and the ways in which evaluation practice may address them, see, for instance, Shapira, 2003.) The problem facing the evaluator is to strike a balance between, on one hand, the increasing cost and diminishing policy relevance of delaying impact assessment and, on the other hand, the need to delay impact assessment to capture potential long-term impacts (Molas-Gallart, Tang, Sinclair, Morrow, & Martin, 1999). In this context, a simpler framework implicitly or explicitly based on a linear model of innovation becomes a requisite when quantitative impact assessments are to be carried out within considerable budget and time strictures.

Despite these pressures, we can find incipient examples of evaluations in which elements of a more complex, theory-led, systemic approach to evaluation are being applied to project or program evaluations. There have been evaluations of Specific Programmes and projects analyzing their effects on the behavior of the social agents involved in the innovation process (behavioral additionality). Luukkonen (1998), for instance, has argued that networking emerges as a major effect of the EC Framework Program initiatives. Several evaluations have focused on determining the network effects of EC-funded R&D programs. This approach has been used to analyze the outcome of specific programs (Laredo, Kahane, Meyer, & Vinck, 1992) or more often to study the impact on participating countries taking part in European programs (Laredo, 1998). The formal techniques of social network analysis have also been used to this end (Sanz

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 77

Menéndez, Fernández Carro, & García, 1999). Referring to different approaches, a study for the EC (Fahrenkrog, Polt, Rojo, Tubke, & Zinocker, 2002) suggests that strategic intelligence tools are now beginning to take hold and play a more prominent role in the EU policy-making process.

These examples, however, fall short of a widespread application of the complex, theory-led approaches to evaluation discussed above. It is then not surprising that experts in the evaluation community are expressing concerns about the way in which STI program evaluations are being carried out in practice. The report we have just mentioned (Fahrenkrog et al., 2002) also argues that the evaluation of research and technological development has tended to focus on projects and programs rather than the wider policy environment and that the practice of policy evaluation should move away from attempts to quantify policy impacts and toward efforts to facilitate learning from previous experiences and supporting decision making and policy strategy definition. Barré (1999) argues that, paradoxically, too narrow a focus on impact measurements creates a risk of reducing credibility and relevance for policy makers. He notes that most policymaking and evaluation activities undertaken in the EU have until recently been based on the traditional linear model and that the growing focus on impact measurement may result in an even narrower approach to evaluation. Pursuing impact measurements, evaluations assume a temporal sequence in which R&D is defined and done and from which measurable society impacts follow. Measurement methods implicitly based on this linear model, Barré argues, will fail to account for the complex interactions between R&D and its applications. Recent efforts to develop evaluation tools and methods have been hampered, in Barré’s view, by the difficulties in building a nonlinear model that can account for the complex chain of interactions and multiple actor dynamics involved in the innovation process while at the same time yielding unequivocal impact measures. Similarly, van Raan (2000) worries that evaluation practice continues to be primarily concerned with the short-term and direct impact of R&D programs rather than the broader implications of STI policy portfolios (van Raan, 2000).

Conclusion: Narrowing the Gap Between Theory and Practice in European STI Policy Evaluation There is, in principle, a strong rationale to support the use of theory-led approaches to evaluation that take into account the systemic nature of the relationship between science, innovation, and social and economic welfare. The systemic perspectives that underpin the proposed intelligent benchmarking and formative evaluation approaches described above cannot be operated in isolation from the rest of the policy process. The way in which they feed into policy definition and implementation are an integral part of the approach. Under a systemic view of the innovation process, policies have to be directed at improving the weakest link or node in the system and would benefit, it has been suggested, from a systemic evaluation involving bottleneck analysis (Arnold, 2004). From this perspective, successful STI policies are to be based on the construction of a policy portfolio rather than the separate application of individual policy instruments and the role of evaluation changes from focusing on ex-post analysis of program or project performance to a broader, more diffuse role. The new approaches reviewed in this article emphasize the formative role of evaluation, contributing to learning and future policy definition, and entail a switch from quantifying specific and attributable policy impacts toward promoting learning to support the formulation of an integrated set of innovation policies (Fahrenkrog et al., 2002).

Yet the progress in this direction has been slow. After a lag of almost a decade, STI policy evaluation is starting to catch up with the latest thinking about how nonlinear and complex innovation processes work. Traditional project and program evaluations coexist with the incipient

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 © 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

78 American Journal of Evaluation / March 2006

use of formative approaches, whereas policy makers and politicians—the final clients of policy evaluations—continue to be primarily concerned with the short-term and direct impact of STI policies. In particular, the political need for impact assessments, preferably quantitative, is, if anything, increasing. The need to justify R&D expenditures against other competing social needs and the diffusion of New Public Management approaches to public administration emphasizing the need for accountability and results-based management drive the political need for evaluations that are able to measure the benefit of specific policy actions in terms of simple economic indicators. Initiatives need to be measured in terms of their effect on, say, productivity, growth, or employment. These outcome measures, required by evaluation clients, can more easily be delivered by evaluation frameworks based, implicitly or explicitly, on linear models of innovation, particularly when taking into account the limited resources that can, in practice, be allocated to evaluation. As Rip (2003) noted, “the evolving state of the art of R&D evaluation is co-produced in the forcefields of concerns and interests of principals commissioning R&D, the subjects of such evaluations, and the evaluation professionals” (p. 49). The problem is that as evaluation professionals begin to favor and actively promote theory-led evaluations, political realities often force evaluators to focus on the development of methods implicitly anchored in outdated linear views of the innovation process.

The tension between the systemic evaluation approaches proposed by evaluation experts and the impact measurements required by clients for accountability purposes is not going to be solved easily. A systemic approach to evaluation is, in principle, capable of yielding impact measurements. Intelligent benchmarking, as discussed above, is one approach that combines a theory-led systemic approach to evaluation with a focus on measurement. It can provide a view of the innovation system and of where the main systemic failures and bottlenecks may lie but cannot establish a direct relationship between a policy measure and a specific outcome measurement of policy performance (like, for instance, the effect of a specific R&D program on aggregate productivity). One practical reason for such limitation is that as we have already discussed, the impacts of STI initiatives are bound to be long term whereas the political need for impact measurements is almost immediate after the completion of a program. Quantifying such impacts through the use of complex modeling is theoretically feasible but in practice would require an intensive, complex, and long-term effort at data gathering and analysis. The main problem here is that we are trying to measure the exact extent to which specific outcomes can be attributed to policy measures, which are likely to play a relatively small role among the many other factors that will emerge in a systemic model. Such detailed attribution requires comprehensive modeling and measurements that are not currently available. Furthermore, because of the delayed impact of many STI policy outcomes, impact measurement cannot provide results that can be used in short-term program evaluation required by policy makers. It is not surprising that impact measurements of STI policy initiatives carried out for evaluation purposes have, so far, been based on simpler linear models that can be operationalized more easily.4 A systemic approach to evaluation, such as the one suggested by Arnold (2004), will yield a complex picture of the context of innovation with an assessment of the role that a program or set of programs have played in modifying it. It may uncover system bottlenecks and the processes through which a specific initiative has had, or has failed to have, the desired impact. Currently, this does not satisfy the political need for clear, unequivocal, and simple measures of policy impact.

The generic lesson of this experience is that the role of the professional community of evaluators is central. When theory-led evaluations are not suited to the delivery of simple performance measurements requested by policy makers, the evaluator faces a predicament. On one hand, to seek short-term influence would require a focus on evaluations that are not based on the evaluator’s best theoretical understanding of the policy impact processes. On the other hand, to

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 © 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 79

insist on the development of theory-led approaches is a more arduous and risky task, as the work of the evaluator may in the end appear to have proved irrelevant. Evaluations oriented to the generation of clear impact measurements based on simpler models may provide relatively easy answers to the accountability problem that the current political environment faces. By providing results that can be more easily digested into policy action, evaluations can, for instance, provide a powerful tool to justify the existence of a program or otherwise cancel it. Instead, whenever the answers provided by theory-led systemic evaluations are complex, for them to have an effect, they will require a sophisticated policy in which to operate, and the use of the evaluation, if any, will be more difficult to determine.

Yet we need to be reminded that the channels through which evaluations can influence the policy process are diverse and complex (Henry & Mark, 2003). Although attempting to push systemic evaluations against the grain of the political need for accountability-oriented assessments may first appear to risk irrelevance, without the efforts to develop and, above all, implement new theory-led approaches to evaluation, there may be a risk that policy definition may be informed by method-led evaluations based on discredited models of the policy impact process.

Whenever there is policy interest in developing evidence-based policy processes, progress toward theory-led evaluations remains. In the area analyzed for this article, European STI policy evaluation, the proposals of the evaluation experts are at least being listened to by EC officials, who have supported different research initiatives to study and develop new approaches to evaluation. The results of these efforts have been broadly disseminated among the evaluation and STI policy communities and, as we have shown, have had an incipient but far-fromdominant effect on evaluation practice. Those in the evaluation community proposing a systemic, theory-led approach to STI policy evaluation are helped by the way in which the EC operates a structured evaluation process revolving around the tendering of evaluation work to external organizations and experts. The leading evaluation experts, who are repeatedly involved in evaluation projects through this system, are often committed to the implementation of more sophisticated, theory-led approaches to evaluation. Although their efforts may at times clash with the requests to develop techniques to yield simpler impact measures, by developing and suggesting ways in which systemic evaluation can be organized and inserted in the policy process, the role of the evaluator becomes formative in the wider sense of the word.

Notes 1. Technology foresight refers to a body of techniques and practices to investigate systematically the future of science and technology developments to identify areas of strategic R&D and emerging technologies (Office of Science and Technology, 1996).

2. Technology assessment attempts to assess the implications of adopting particular technological options and to describe the potential social and economic impacts of technologies on society (Meyer-Krahmer & Reiss, 1992).

3. A description of the way the Framework Programmes and their evaluations operate can be found in a comprehensive review of Framework Programme evaluations carried out by several European organizations (Policy Research in Engineering Science and Technology [PREST] et al., 2002).

4. See, for instance, the models recently developed by the EC-funded NO-REST project to assess the impact of standardization policies. This project represents a good example of a multilayered approach using different methodologies to build a comprehensive view of the innovation process and the effect of standardization on it. Yet the quantitative component of this initiative is based on comparatively simple and linear models (see http://www.no-rest.org).

References Arnold, E. (2004). Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation, 13(1), 3-17.

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

80 American Journal of Evaluation / March 2006 Barré, R. (1999). Public research programmes: Socio-economic impact assessment and user needs. IPTS Report Special Issue: Evaluation and Research Activities, 20, 5-9.

Betz, F. (1994). Basic research and technology transfer. International Journal of Technology Management, 9(5, 6, 7), 784-796.

Boden, R., Gummett, P., Cox, D., & Barker, K. (1998). Men in white coats. . . . Men in grey suits: New Public Management and the funding of scientific research in the UK. Journal of Auditing, Accounting and Accountability, 11(3), 267-291.

Bush, V. (1945). Science, the endless frontier (a report to the president by Vannevar Bush, director of the Office of Scientific Research and Development). Washington, DC: U.S. Government Printing Office.

Campbell, D., & Julian, S. (1963). Experimental and quasi-experimental designs for research on teaching. Chicago: Rand McNally.

Dalpe, R., & Anderson, F. (1993). Evaluating the industrial relevance of public R&D laboratories. In B. Bozeman & J. Melkers (Eds.), Evaluating R&D impacts: Methods and practice (1st ed., pp. 207-228). Norwell, MA: Kluwer Academic.

Dietz, J. S. (2003). Assessing RTD program portfolios in the European Union. In P. Shapira & S. Kuhlmann (Eds.), Learning for science and technology policy evaluation. experiences from the United States and Europe (pp. 204222). Cheltenham, UK: Edward Elgar.

Fahrenkrog, G., Polt, W., Rojo, J., Tubke, A., & Zinocker, K. (Eds.). (2002). RTD evaluation toolbox: Assessing the socio-economic impact of RTD policies. Seville, Spain: Joint Research Centre, European Commission.

Feller, I. (2002). Performance measurement redux. American Journal of Evaluation, 23(4), 435-452.

Freeman, C., & Soete, L. (1997). The economics of industrial innovation. Cambrdige, MA: MIT Press.

Georghiou, L. (1995). Assessing the Framework Programmes—A meta-evaluation. Evaluation, 1(2), 171-188.

Georghiou, L. (2003). Evaluation of research and innovation policy in Europe—New policies, new frameworks? In P. Shapira & S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 65-79). Cheltenham, UK: Edward Elgar.

Georghiou, L., Ribgy, J., & Cameron, H. (2002). Assessing the socio-economic impacts of the Framework Programme.

Manchester, UK: Policy Research in Engineering Science and Technology (PREST), University of Manchester.

Gibbons, M., & Georghiou, L. (1987). Evaluation of research: A selection of current practices. Paris: Organization for Economic Co-operation and Development.

Grupp, H. (2000). R&D evaluation. Research Evaluation, 8(1), 87-99.

Guy, K. (2003). Assessing RTD program portfolios in the European Union. In P. Shapira & S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 174-203).

Cheltenham, UK: Edward Elgar.

Guy, K., & Arnold, E. (1998). Strategic options for the evaluation of the R&D Programmes of the European Union (final report). Brussels, Belgium: European Parliament, Directorate General for Research, Scienctific and Technological Options Assessment.

Guy, K., & Nauwelaars, C. (2003). Benchmarking STI policies in Europe: In search of good practice. Institute for Prospective and Technological Studies Report, 71, 20-27.

Hawkins, R., Montalvo-Corral, C., van Audenhove, L., & Molas-Gallart, J. (2004). Analysis via case studies of the instruments used in the IST programme, submitted to European Commission, DG Information Society, Evaluation and Monitoring (017.31126/01/01).

Henry, G., & Mark, M. (2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24(3), 293-314.

Hobday, M. (1988). Evaluating collaborative R&D programmes in information technology: The case of the U.K. Alvey programme. Technovation, 8, 271-298.

Jaffe, A. B. (1998). Measurement issues. In L. M. Branscomb & J. H. Keller (Eds.), Investing in innovation (pp. 64-84).

Cambridge, MA: MIT Press.

Karatzas, I., & Fayl, G. (1999). Editorial. Institute for Prospective and Technological Studies Report Special Issue: Evaluation and Research Activities, 40, 2-4.

Kastrinos, N. (1994). Evaluating the impact of the EC Framework Programme. Technovation, 14(10), 679-688.

Kline, S. J. (1985). Innovation is not a linear process. Research Management, 28(4), 36-45.

Kline, S. J., & Rosenberg, N. (1986). An overview of innovation. In R. Landau & N. Rosenberg (Eds.), The positive sum strategy: Harnessing technology for economic growth (pp. 275-305). Washington, DC: National Academy Press.

Kostoff, R. N. (1997). The handbook of research impact assessment (No. DTIC ADA296021). Arlington, VA: Office of Naval Research.

Kuhlmann, S. (1999). Distributed intelligence: Combining evaluation, foresight and technology assessment. Institute for Prospective and Technological Studies Report, 40, 16-22.

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 © 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

Molas-Gallart, Davies / Toward Theory-Led Evaluation 81 Kuhlmann, S. (2003). Evaluation as a source of ‘strategic intelligence.’ In P. Shapira & S. Kuhlmann (Eds.), Factors affecting technology transfer in industry-US federal laboratory partnerships (pp. 352-375). Cheltenham, UK: Edward Elgar.

Laredo, P. (1998). The networks promoted by the framework programme and the questions they raise about its formulation and implementation. Research Policy, 27(6), 589-598.

Laredo, P., Kahane, B., Meyer, J. B., & Vinck, D. (1992). The research networks built by the MHR4 Programme.

Brussels, Belgium: Commission of the European Communities.

Leeuw, F. L. (2003). Reconstructing program theories: Methods available and problems to be solved. American Journal of Evaluation, 24(1), 5-20.

Lundvall, B.-A. (1992). National systems of innovation: Towards a theory of innovation and interactive learning (1st ed.). London: Pinter.

Luukkonen, T. (1998). The difficulties in assessing the impact of EU framework programmes. Research Policy, 27(6), 599-610.

Luukkonen, T. (2002). Research evaluation in Europe: State of the art. Research Evaluation, 11(2), 81-84.

Mayne, J. (2001). Attribution through contribution analysis: Using performance measures sensibly. Canadian Journal of Program Evaluation, 16(1), 1-24.

Meyer-Krahmer, F., & Reiss, T. (1992). Ex ante evaluation and technology assessment—Two emerging elements of technology policy evaluation. Research Evaluation, 2(1), 47-54.

Molas-Gallart, J., Salter, A., Patel, P., Scott, A., & Duran, X. (2002). Measuring third stream activities. Brighton, UK: SPRU. Molas-Gallart, J., Tang, P., Sinclair, T., Morrow, S., & Martin, B. (1999). Assessing research impact on non-academic audiences. Brighton, UK: SPRU. Mowery, D. C., & Rosenberg, N. (1979). The influence of market demand upon innovation: A critical review of some recent empirical studies. Research Policy, 8, 105-153.

O’Doherty, D., & Arnold, E. (2003). Understanding innovation: The need for a systemic approach. Institute for Prospective and Technological Studies Report, 71, 29-36.

Office of Science and Technology. (1996). Winning through foresight. A strategy taking the foresight programme to the millenium. London: Department of Trade and Industry.

Organization for Economic Co-operation and Development. (2002). Benchmarking industry-science relationships.

Paris: Author.

Policy Research in Engineering Science and Technology (PREST). (1990). The impact and utility of European Commission research programme evaluation reports (EUR13098 EN). Brussels, Belgium: European Commission.

Policy Research in Engineering Science and Technology (PREST), AUEB, BETA, ISI, Joanneum Research, IE HAS, et al. (2002). Assessing the socio-economic impact of the framework programme. Manchester, UK: University of Manchester.

Rip, A. (2003). Societal challenges for R&D evaluation. In P. Shapira & S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 32-53). Cheltenham, UK: Edward Elgar.

Roessner, J. D. (2002). Outcome measurement in the USA: State of the art. Research Evaluation, 11(2), 85-93.

Rothwell, R. (1992). Successful industrial innovation: Critical factors for the 1990s. R&D Management, 22(3), 221239.

Rothwell, R. (1993, May). Systems integration and networking: The fifth generation innovation process. Paper presented at the Chaire Hydro-Quebec Conference en Gestion de al Technologie, Montreal, Quebec, Canada.

Sanz Menéndez, L., Fernández Carro, J. R., & García, C. E. (1999). Centralidad y cohesión en las redes de colaboración empresarial en la I+D subsidiada [Centrality and cohesion in interfirm collaborative networks participating in publicly funded R&D programs]. Papeles de Economía Española, 81, 219-241.

Schmookler, J. (1966). Invention and economic growth. Boston: Harvard University Press.

Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagné, & M. Scriven (Eds.), Perspectives of curriculum evaluation (pp. 39-83). Chicago: Rand McNally.

Shapira, P. (2003). Evaluating manufacturing extension services in the United States: Experiences and insights. In P. Shapira & S. Kuhlmann (Eds.), Factors affecting technology transfer in industry-US federal laboratory partnerships (pp. 260-292). Cheltenham, UK: Edward Elgar.

Sherwin, C. W., & Isenson, R. S. (1967, March). Estimating the science and technology components of an R&D budget.

Paper presented at the Second Cost-Effectiveness Symposium of the Washington Operations Research Council, Washington, DC. Soete, L., & Corpakis, D. (2003). Editorial: R&D for competitiveness and employment—The role of benchmarking.

Institute for Prospective and Technological Studies Report, 71, 2-12.

Stame, N. (2004). Theory-led evaluation and types of complexity. Evaluation, 10(1), 58-76.

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 © 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.

82 American Journal of Evaluation / March 2006 STRATA-ETAN Expert Working Group. (2002). Benchmarking national research policies: The Impact of RTD on Competitiveness and Employment (IRCE). Brussels, Belgium: European Commission Directorate General Research. van Raan, A. F. J. (2000). R&D evaluation at the beginning of the new century. Research Evaluation, 8(2), 81-86. von Hippel, E. (1988). The sources of innovation. New York: Oxford University Press.

Downloaded from http://aje.sagepub.com by Juan Pardo on March 20, 2008 Š 2006 American Evaluation Association. All rights reserved. Not for commercial use or unauthorized distribution.