Introduction

The new research ‘impact’ agenda is likely to have a profound effect on the social science research community in wide-ranging ways, shaping the sorts of research questions and methods scholars are selecting, their networks and collaborations, as well as changing institutional structures of support within higher education institutions. Yet concepts and models for defining and measuring impact have been subject to surprisingly little social scientific scrutiny. While there is an extensive literature on research-policy relations across fields of social science (notably in sociology, science and technology studies, social policy, political science and public management), only a very narrow range of these contributions have been marshalled to develop guidance and practice on ‘impact’. Indeed, prevalent guidelines and models are frequently based on surprisingly simple and linear ideas about how research can be ‘utilised’ to produce more effective policies (Smith and Stewart, 2016).

In this article, we seek to advance the debate on impact by setting out four different approaches to theorising research-policy relations, drawn from wider social science literature. Each set of theories is categorised according to its core assumptions about the inter-relations between the two spheres. The first approach focuses on a ‘supply’ model of research-policy relations, examining how knowledge and ideas shape policy. The second challenges the idea that research is independent of politics and policy, instead focusing on how political power shapes knowledge. The third approach takes this line further, suggesting that research knowledge and governance are co-produced through an ongoing process of mutual constitution. And the fourth approach offers a radically contrasting account, suggesting that there is no overarching causality between science and politics, but that politics only selectively appropriates and gives meaning to scientific findings. Figure 1 offers a simple representation of these four ways of modelling the relations.

Fig. 1
figure 1

Research-policy relations

Full size image

This figure represents in visual form the direction of influence between research, expert knowledge and science; and policy and politics. The first panel represents theories assuming that research shapes policy. The second panel depicts the idea that policy and politics shape the production of research. In the third panel, the circular arrows convey the idea of research and policy being mutually constitutive. While the fourth panel suggests that there is no direct causal relationship between research and policy, but that instead, the two ‘systems’ only selectively pick up on signals from the other system.

This four-way schema offers a useful resource in two main ways. First, it offers a classificatory tool for mapping, comparing and analysing a range of often disparate theoretical approaches in the emerging field of knowledge-policy relations–theories that emanate from a wide set of social science disciplines, and are informed by quite divergent assumptions about knowledge and governance. The second, more applied, use of the schema is to identify the plurality of ways of conceptualising knowledge-policy relations. In doing so, we demonstrate that prevalent models of impact are based on one particular set of assumptions about the role of research in policy, and not necessarily the most theoretically sophisticated at that. By briefly setting out each of the four sets of theories, we show how each is based on quite distinct assumptions about knowledge and policy, and that each has different implications for how we might go about defining and measuring impact.

The ‘impact’ agenda in UK research funding

The emphasis on ‘research impact’ has been increasing steadily across a number of OECD countries over the past decade, notably Australia (Donovan, 2008; Chubb and Watermeyer, 2016), Canada (Canadian Academy of Health Sciences CAHS, 2009), the Netherlands (Mostert et al., 2010) and the USA (Grant et al., 2010) but the influence of this agenda is particularly pronounced in the UK, which can be seen as something of a pioneer in implementing these approaches (see Bornmann, 2013 and Grant et al., 2010 for useful comparative overviews). There are currently two major incentives for social scientists in the UK to demonstrate that their research influences policy. First, the national appraisal mechanism for assessing university research (which informs decisions about the distribution of core research funding), known as the Research Excellence Framework (REF), has begun awarding 20% of overall scores to institutions on the basis of case studies of research impact (UK higher education funding bodies 2011). Second, accounts of the work that will be undertaken to achieve research impact (‘pathways to impact’) now form a significant section of grant application processes for the UK funding councils (Research Councils UK, Undated). The upshot is that obtaining core research funding and project-specific grants from publicly funded sources in the UK are now strongly dependent on researchers’ abilities to respond adequately to questions about the non-academic value of their work (Smith and Stewart, 2016).

The current focus on ‘research impact’ reflects a longer-standing concern with the societal return on public funding of science (Brewer, 2011; Clarke, 2010). This agenda was given particular impetus by New Labour government commitments to taking a more ‘evidence-based’ approach to policymaking (Labour Party, 1997), with official statements evoking a simple, linear conceptualisation of the relationship between research and policy (e.g., Cabinet Office, 1999, 2000; Blunkett, 2000). It is this kind of thinking that appears to have shaped tools and guidance on impact (Smith, 2013a). Indeed, while different public bodies have adopted a variety of models, RCUK and REF advisory documents tend to share a number of common features (AHRC, 2014, 2015; ESRC, 2014a, 2014b, 2014c; MRC, 2014; Research Councils UK, Undated): (i) a consensus that researchers have a responsibility to articulate the impact of their research to non-academic audiences; (ii) an assumption (most explicit in the REF impact case studies) that this impact can be documented and measured; (iii) a belief that the distribution of research funding should (at least to some extent) reflect researchers’ ability to achieve ‘impact’; and, following from this, (iv) an expectation that researchers’ own efforts to achieve research impact will play a significant role in explaining why some research has impact beyond academia and some does not.

This approach is exemplified in HEFCE’s template for REF2014 impact ‘case studies’ (REF, 2014, 2011). The template calls for an account of the ‘underpinning research’ that exerted impact, implying that impact is achieved through policy-makers adjusting their beliefs in response to clearly delineated research findings. The implication is that research findings are created independently of policy or politics: research is treated as an exogenous variable that feeds into policy-making. Secondly, such findings are expected to have been published as ‘outputs’ that are rated 2*, or ‘nationally leading in terms of their originality, significance and rigour’ (REF2014, 2014). Thus a clear link is posited between the quality of research and the desirability of rewarding impact: impactful research should meet a certain quality threshold. Thirdly, researchers are required to chart how their findings came to exert impact, and to provide evidence to corroborate their claims. Evocative of the ‘pathways to impact’ section of RCUK grant proposals (Research Councils UK, Undated), this requirement implies that researchers can trace the effects of their work through describing a series of concrete activities and information flows – events, meetings, media coverage, and so on.

There is currently no agreed way of tracking research impacts and, in this context, some academics have identified more specific frameworks and approaches, including the ‘payback framework’ (Donovan and Hanney, 2011) and the ‘research contribution framework’ (Morton, 2015). However, others have criticised the simplistic and linear conceptualisations of research-policy relations that appear to underpin the UK’s overarching approach to research impact, particularly those with in-depth knowledge of the policy process and/or the relationship between research and policy (Greenhalgh et al., 2016; Smith and Stewart, 2016). Theories of public policy have shown that policy-making rarely occurs in such neat sequential stages (Cairney, 2016), and that evidence often plays a rather limited role in decision-making (Boswell, 2009a). In the context of such criticisms and concerns, we consider the rich body of literature from political science, policy studies, sociology of knowledge, and science and technology studies, which has informed understandings of the complex relationship between knowledge and policy. Drawing on this wider literature, we now set out four different approaches to theorising the relationship, and consider their implications for the impact agenda.

Four approaches to conceptualising research-policy relations

Knowledge shapes policy

A range of theories and models of the relationship between academic knowledge and policy were developed by US and UK scholars in the 1970s and 1980s (Blume, 1977; Caplan, 1979; Rein, 1980; Weiss, 1977, 1979). Notably, a number of contributions produced ‘instrumental’ models of knowledge utilisation (see Weiss, 1979 for an overview), according to which knowledge either ‘drives’ policy, or policy problems stimulate research to provide direct solutions (again, see Weiss, 1979). Much of the work undertaken in the 1970s and 1980s demonstrated that while there are occasional examples of research feeding into policy in this manner, such simple models failed to capture the intricacies of the interactions between research and policy (Rein, 1980; Weiss, 1979). Yet, it was precisely these simple, instrumental notions of the role of research in policy that seem to have become increasingly embedded within UK policy, including higher education policy, leading Parsons to reflect that the Labour government’s commitments to ‘evidence-based policymaking’ marked:

not so much a step forward as a step backwards: a return to the quest for a positivist yellow brick road leading to a promised policy dry ground-somewhere, over Charles Lindblom - where we can know ‘what works’ and from which government can exercise strategic guidance. (Parsons, 2002, p 45)

Understandably, official commitments to employing evidence in a direct, linear sense triggered a raft of assessments of the extent to which particular policies do reflect the available evidence. Perhaps unsurprisingly, most of these found the government’s use of evidence has been highly selective (e.g., Boswell 2009a, 2009b; Katikireddi et al., 2011; Naughton, 2005; Stevens, 2007) and this, in turn, has triggered renewed interest in two, more complex models of the ways in which research knowledge shapes policy, each of which has very different implications for the research impact agenda.

The first of these approaches seeks to address what is perceived as a ‘gap’ between the research and policy communities. On this account, research has the potential to be highly relevant to policy, but its impact is often reduced by problems of communication. Research may not be disseminated in a form that is relevant or accessible to policy-makers; or officials have insufficient resources to process and apply research findings. For example, Lomas (2000) and Lavis (2006) both underline the importance of achieving shared understandings between researchers and policymakers, arguing that increased interaction between the two groups will improve the use of research in policy. These authors tend to assume that research would be more frequently employed by policymakers if only they could better access and understand the findings and if the findings were of relevance. Thus the focus is on improving the mechanisms of communication, and the levels of trust, between researchers and policymakers. A stronger version of this ‘gap’ account posits that this reflects a deeper cultural gap between researchers and policy actors. Thus Caplan (1979) suggests that these actors should be seen as distinct ‘communities’ guided by different values and beliefs–a notion we discuss further in the fourth set of theories, considered later in the paper.

The weaker version of this ‘gap’ approach, however, suggests that there are various practical steps that can be taken to improve the flow of knowledge from research to policy. Indeed, several reviews of knowledge transfer provide practical recommendations for researchers seeking to influence policy (Contandriopoulos et al., 2010; Innvaer et al., 2002; Mitton et al., 2007; Nutley et al., 2003; Oliver et al., 2013; Walter et al., 2005), suggesting researchers should ensure research is accessible, by providing clear, concise, timely summaries of the research, tailored to appropriate audiences; and develop ongoing, collaborative relationships with potential users to increase levels of trust and shared definitions of policy problems and responses. In structural terms, the findings of these reviews call for improved communication channels, via ‘knowledge broker’ roles and/or knowledge transfer training and sufficiently high incentives for researchers and research users to engage in knowledge exchange. Of the various conceptualisations of the relationship between research knowledge and policy, it is this way of thinking which appears to have had most influence on current approaches to incentivising research impact in the UK. As we shall see, however, the approach is widely criticised by the alternative theories of research-policy relations we explore later in the article.

A second popular theory of how research shapes policy emerges from Weiss’ (1977, 1979) notion of the ‘enlightenment’ function of knowledge in policymaking. This account proposes that knowledge shapes policy through diffuse processes, resulting from the activities of various, overlapping networks, which contribute to broader, incremental and often largely conceptual changes (Hird, 2005; Walt, 1994). Radaelli’s (1995) notion of ‘knowledge creep’ is one of several more recent conceptualisations to build on this idea, and we can find similar assumptions in ideational theories of policy change (Béland, 2009; Hall, 1993; Schmidt, 2008). The implication of these accounts is that research influences policy over long periods through gradual changes in actors’ perceptions and ways of thinking (an idea that is also evident in theories of co-production, as discussed later) rather than through immediate, direct impacts. Whilst this body of work does not discount the possibility that research might contribute to what eventually become significant shifts in policy approaches, it suggests that assessments aiming to trace the impact of research on particular policy outcomes are likely to miss a potentially broader, more diffuse kind of conceptual influence.

The implications of this way of conceptualising the relationship between academic knowledge and policy for ideas about research impact are more challenging (indeed, the ‘enlightenment’ model has been criticised by some scholars seeking to improve the use of evidence in policy for its lack of practical utility (Nutley et al., 2007)). Taking the more conceptual influence of research seriously suggests that incentives for achieving impact ought to shift away from individual researchers and projects to consider how to support the collective diffusion of much more diverse (potentially interdisciplinary) bodies of work. Given that multiple authors are likely to be involved, and that various factors unrelated to the underpinning research (or its communication) are likely to inform when and how knowledge shapes policy, it seems to make little sense to reward individual researchers (or even teams of researchers) for ‘achieving’ research impact. Instead, research impact might be supported by encouraging groups of researchers to work together on developing policy messages from diverse studies on particular policy topics (or, to support knowledge brokers to do this kind of work).

This is a very different model from both the RCUK pathways to impact approach, which encourages individual researchers or research teams to try to achieve research impacts on the back of single studies, and the REF impact case study approach, which encourages single institutions to narrate stories of impact based solely on the work of researchers they employ. Indeed, recent assessments of the REF impact case study approach have specifically highlighted the tendency not to adequately support these kinds of synthesised approaches to achieving impact (Manville et al., 2015; Smith and Stewart, 2016). For the moment, while some of the guidance documents relating to the UK impact agenda do acknowledge conceptual forms of influence, the mechanisms for monitoring and rewarding impact seem preoccupied with ‘instrumental’ research impact achieved on the back of research undertaken by individual researchers or small groups within single institutions.

Politics shapes knowledge

Perhaps the most obvious critique of the ‘knowledge shapes policy’ model reverses this relationship to highlight the various ways in which policies and politics shape knowledge and the use of knowledge. There is a rich body of literature theorising how state-building and modern techniques of governance have shaped the production of social knowledge (Foucault, 1991, Heclo, 1974; Rueschemeyer and Skocpol, 1996), as well as how power relations are implicated in the construction of expert authority (Gramsci, 2009). What these diverse contributions share is the notion that an underlying political project is driving research production and utilisation, whether that project is the production of self-regulating subjects (as some Foucauldian interpretations suggest) or the continuing dominance of ruling elites and ideologies (as Gramscian analyses tend to posit). From this perspective, research utilisation in policymaking is understood as profoundly constrained; whilst those involved in the construction of policy are not necessarily consciously aware of the forces shaping their decisions, any attempt to engage with research must be understood as part of a wider political project. At the very least, such analyses suggest that only research that can be used to support these dominant ideas and interests will be employed in policymaking, while research that challenges dominant ideas will be discounted (see Wright et al., 2007). A stronger interpretation would hold that the research process is itself shaped by the ‘powerful interests’ directing policy agendas (e.g., Navarro, 2004).

The more applied literature concerning the relationship between research and policy also provides examples of this way of thinking about the relationship. In her overview of various ‘models’ of the relationship between research and policy, Weiss, for example, describes what she calls the ‘political model’, where research is deployed to support pre-given policy preferences; as well as a ‘tactical model’, where research is used as a method of delaying the decision-making process, providing policymakers with some ‘breathing space’ (Weiss, 1979). In the first case, the research process itself is not necessarily informed by politics but the decision to employ research (or not) is entirely political. In other words, political ideology and/or more strategic party politics inform the ways in which political actors respond to research evidence (e.g., Bambra, 2013). In the second, the commissioning of research might itself be understood as a political act (or, at least, an act that creates political benefits–see Bailey and Scott‐Jones 1984). In either case, efforts to reward researchers for ‘achieving’ research impact would seem misplaced.

The extent to which politics can shape research is perhaps most overt in research that is directly commissioned by sources with particular political/policy interests; reviews have repeatedly demonstrated that research funded by commercial sources, such as the pharmaceutical (e.g., Lundh et al., 2012) and tobacco industries (e.g., Bero, 2005), is more likely to present findings that are useful to those interests (see also Bailey and Scott‐Jones, 1984). In other contexts, it has been suggested that researchers may struggle to maintain their independence where research is commissioned directly, or indirectly, by government sources (e.g., Barnes, 1996; Smith, 2010). This kind of political influence may be felt both overtly and subtly, with researchers responding to signals from research funders as to what is likely to be funded (and what is not), what they are hoping (or expecting) to be found and what they are not (Knorr-Cetina, 1981; Smith, 2010), as we discuss further in the following section.

A second group of theories which call attention to the way in which politics can shape knowledge focus on the impact of institutions and organisational structures on policymaking and research. Similar to the previous group of theories, such accounts assume that the wider structures in which actors are located are key to explaining policy outcomes. Whilst the more political accounts discussed above highlight the ways in which power relations and elite interests can shape research and its use, these theories focus on organisational and decision-making structures. The most well-known of such theories are the various forms of institutionalism, of which ‘historical institutionalism’ is one of the most widely employed forms (see Immergut, 1998 for an overview). From this perspective, rather than constituting the collective result of individual preferences, policy processes (including efforts to engage with research) are considered to be significantly shaped by the historically constructed institutions and policy procedures within which they are embedded (Immergut, 1998).

Those who have contributed to the development of this genre of work have emphasised that such theories do not suggest that particular policy outcomes are inevitable–and indeed, as we discussed in the previous sections, under certain conditions existing paradigms can be superseded by new ideas, leading to substantial policy change (Hall, 1993). However, such theories do suggest that it becomes increasingly difficult to change the overall direction of a policy trajectory as previous decisions become ever more deeply embedded in institutional structures and ways of thinking (e.g., Kay, 2005). Employing these kinds of theories, Smith (2013b) has demonstrated how the institutionalisation of particular ideas about health and economic policy function as filters to research-based ideas about health inequalities, encouraging those ideas that support existing institutionalised ideas (or ‘policy paradigms’) to move into policy, while blocking or significantly transforming more challenging ideas.

This way of thinking about the relationship between knowledge and policy suggests that research is constantly being influenced by policy and politics and that efforts to bring researchers and policymakers closer together are like to exacerbate this in ways that may not be desirable. At best, from this perspective, the research impact agenda seems likely to reward some academics (and not others) for achieving impacts that had far more to do with political interests and agendas than the research or impact activities of those academics. At worst, the impact agenda will lead to the increasing politicisation of research (and an associated reduction in academic freedom). Indeed, some of the most critical responses to the impact agenda are informed by these kinds of concerns. Cohen (2000) and Hammersley (2005), for example, have warned that the restrictions being placed on publicly-funded research to be ‘useful’ to policy audiences is limiting the potential for academics to promote ideas that are out-of-line with government policies. Likewise, Davey Smith et al., (2001), argue that efforts to achieve evidence-based policy may, in fact, do more to stimulate research that is shaped by policy needs than to encourage better use of research in policy-making.

Co-production

A third way of theorising research-policy relations has emerged from science and technology studies (STS), and posits a much more complex inter-relationship between knowledge production and governance. This approach is encapsulated in the idea of ‘co-production’: the claim that knowledge and governance are mutually constitutive (Jasanoff, 2004).

Similar to the approaches discussed in the last section, such accounts see knowledge as profoundly shaped by politics. But the notion of co-production focuses not just on the social and political constitution of science. It is also attentive to the other direction of influence: the ways in which governance is itself constituted by scientific knowledge. So rather than limiting its attention to how politics shapes knowledge, the notion of co-production posits that scientific and expert knowledge contribute to the construction of political reality (an idea that is, in some ways, simply a stronger version of Weiss’ (1979) account of the enlightenment function of research, discussed earlier). Knowledge provides the concepts, data and tools that underpin our knowledge of social and policy problems and appropriate modes of steering (Voß and Freeman, 2016). Sheila Jasanoff (2004) is arguably the most influential exponent of this approach. In her book States of Knowledge, she explores how knowledge-making is an inherent part of the practices of state-making and governance. States ‘are made of knowledge, just as knowledge is constituted by states’ (Jasanoff, 2004, p 3). Moreover, STS scholars have shown how science does not just produce knowledge and theories that help define social problems and appropriate responses. It also produces skills, machines, instruments and technologies that are deployed in governance (Pickering, 1995).

An important concept informing this approach is that of performativity. This is the idea that social enquiry and its methods are ‘productive’: rather than simply describing social reality, they help to make or enact the social world (Law and Urry, 2004). Indeed, social science needs to be understood as fundamentally embedded in, produced by, but also productive of the social world (Giddens, 1990). Social science thus has effects–it creates concepts and labels, classifications and distinctions, comparisons and techniques that transform the social world. Such concepts and techniques can also help bring into existence the social objects they describe. Osborne and Rose (1999) illustrate this idea with the case of public opinion, a social phenomenon that was effectively created in the 1930s through the emergence of new methods of polling and survey analysis, and is now thoroughly normalised as an object of social scientific enquiry. Similarly, Donald MacKenzie (2006) has explored the performativity of economic models, showing how the theory of options shaped practices in trading and hedging in the financial sector from the 1970s onwards. Similar ideas have been explored by Colin Hay (2007) in his discussion of political disaffection. He argues that public choice theory has contributed to the ‘marketisation’ of party politics, implying that such theories have been ‘performative’ (although he does not use this term).

Theories of co-production also show how science can produce social problems. Through its various scientific and technical innovations, science does not simply solve governance problems, but it also creates new ones (Jasanoff, 2004). The frantic pace of development and progress in science and technology produce a continuous stream of new problems and solutions, which governments often struggle to keep pace with. So new research does not just offer ways of ordering the social world, but can also destabilise existing structures and modes of governance. In areas of policy that are highly dependent on technology and science–such as energy, health, agriculture or defence - policy develops almost in pursuit of science, in an attempt to catch up with, harness and regulate the new technologies and practices it has produced. Thus science creates the very problems that need to be addressed through political intervention (Beck, 1992). The demand for ever more problem-solving knowledge is effectively built into the structure of policy-research relations.

What implications do these approaches have for defining and measuring impact? First, they suggest that we cannot neatly disentangle processes of knowledge production from those of governance. This is not merely an epistemological question–a challenge of finding the right methods or observational techniques to allow us to separate out how social scientific findings have influenced politics or policy (although this is of course difficult to do). It represents a more fundamental ontological problem, in that social scientific knowledge is co-constitutive of politics. Imagine, for example, trying to chart the ‘impact’ of public choice theories on politics. We would not only face the methodological challenge of charting the subtle and incremental processes through which a wide variety of social actors (including politicians, campaigners, lobbyists and the media) appropriated public choice theories about political agency. We would also need to understand the ongoing feedback effects through which such ideas brought about shifts in the behaviour of these actors, in turn gradually transforming political behaviour. If we accept the possibility of such effects, then we need to also consider how such shifts may in turn validate the theories that originally produced them, enhancing their authority and influence. The relationship between social science and politics in this example is one of continuous mutual influence and reinforcement.

Second, the notion of co-production suggests that social science may itself produce social problems that require political responses. Studies of public opinion offer a good example of this. A survey of public attitudes may ‘discover’ unarticulated claims and preferences, which produce new demands for political action. In 2014, Jeffery et al., (2014) found a strong desire on the part of the English respondents they surveyed for institutions that better represented and articulated ‘English’ views. This could be charted as ‘impact’ insofar as the findings of the survey were picked up by politicians and influenced claims-making about UK constitutional reform (and indeed it was submitted as a case study to REF2014). But the research can also be understood as producing a new set of political problems. It encouraged a number of survey respondents to articulate a set of preferences which may previously have been nascent or unspecified. These preferences were then presented as a collective and coherent political claim, which in turn implied the need for enhanced political representation and constitutional reform. Research thus contributed to the construction of a new social problem requiring a political response. As with the case of public choice theory, we can also posit a feedback effect, whereby the social and political adjustments generated by the research might in turn further validate the findings. As politicians sought to represent and mobilise these preferences, this created further political expectations and demands, thereby substantiating the initial research claim that the English desire their own institutions.

One implication of this account is that REF or HEFCE models do not do justice to the more pervasive (but often subtle) influence of social science on policy. Another is that they overlook the feedback effects described above, whereby the political adjustments enacted through social science in turn validate (or possible discredit) the authority of research findings or methods. And a third is that they may actively encourage forms of interference that create more problems than they solve. Policy impact may not always be benign, as we noted earlier.

Assuming we accept such impacts as desirable, how might these processes of co-production be best captured and accredited? They would require quite resource-intensive methodologies, as well as forms of expertise that are not necessarily available across disciplines. Each case study would effectively be a social scientific project in its own right, explored though a range of qualitative and quantitative methods, such as ethnography (as Baim-Lance and Vindrola-Padros, 2015, argue in more detail) process tracing, discourse analysis, interviews and surveys. It is hard to imagine sufficient resource being available for such indepth enquiry, or, indeed, for buy-in to such models and methodologies from across (non-social science) disciplines.

Autonomous spheres

Our final approach to theorising research-policy relations understands science and politics as distinct spheres, each operating according to a separate logic and system of meaning. As we saw earlier, one version of this account is Caplan’s (1979) ‘two communities’ thesis, which identifies a ‘cultural gap’ between researchers and policymakers. This conceptualisation has been subject to a range of critiques, not least, as Lindquist (1990) points out, the fact that this way of thinking about the relationship excludes a range of potentially important actors, such as journalists, consultants and lobbyists. Despite this, whilst not always referring to Caplan’s (1979) work directly, many contemporary assessments of the limited use of research in policy and practice frequently mirror Caplan’s observations by highlighting perceived ‘gaps’ between researchers, policymakers and/or practitioners as a fundamental barrier to the use of research.

In this section, we focus on a more radical account of this ‘gap’, associated with the systems theory of German sociologist Niklas Luhmann (e.g., Luhmann, 1996). On a Luhmannian systems theory account, science and politics are both understood as self-referential or ‘autopoietic’ systems. Although mutually dependent in important ways (they could not survive in a recognisable form without one another), each operates according to its own logic or ‘communicative code’, which determines which communications are relevant to the system. There is no causality or direct influence across systems: rather, operations in one system are selectively perceived and given meaning according to the codes and logics of another system. Thus it does not make sense to conceive of flows, diffusion or causality across systems, and STS concepts such as ‘performativity’ or ‘co-production’ need to be carefully re-specified in terms of how one system ‘models’ and responds to the operations of another.

Luhmann understands the primary building blocks of modern society not as individuals or groups, but as functionally differentiated social systems. Modern societies are increasingly sub-divided into specialised, self-referential systems such as education, health, economy, religion, welfare, science or politics. Each of these systems operates according to its own distinct codes, programmes, logic and mode of inclusion. Unlike on Caplan’s account, these systems are not distinguished in terms of members or institutions. Systems do not consist of discrete groups of people, indeed one person or one organisation can participate in several different systems. However, systems are distinguished in terms of sets of differentiated roles and activities. Each system retains its distinctiveness through developing its own criteria of selection, which help it reduce complexity by only selecting those communications which are relevant to the system.

On this account, science and politics are separate function-systems. Science (including social science) operates according to a binary code of true/false. In other words, it defines relevant communication based on whether it is concerned with establishing truth claims. The system of politics, meanwhile, selects relevant communication on the basis of the binary code of government/opposition. The political system selects and gives meaning to communication based on its relevance to the pursuit of political power and the capacity to adopt collectively binding decisions. At first sight, this seems to be a very narrow way of conceiving social systems. For example, scientists are not just preoccupied with validating truth claims; they are clearly also concerned with winning grants, enhancing their academic reputation, or influencing government policies. But these preoccupations are characterised as participating in different systems. For example, a public funding decision has a distinct meaning and relevance in the systems of science, politics and the economy.

From this perspective, there can be no overarching causality operating between two systems, although it is easy to see how appealing such causal attributions might be to observers. To be sure, one event can have effects across different systems. A government research grant has meaning for both the system of politics and that of science. Yet As Luhmann puts it, the ‘preconditions and consequences of events differ completely according to system reference’, and observers should not ‘cross-identify events over boundaries’ (Luhmann, 1991, p 1438). Instead, Luhmann conceives of the relationship as highly selective connections between systems and their environments. Systems that are reliant on other systems in their environment develop models, or assumed regularities, to help them keep tabs on the other system. For example, science will develop a certain way of observing and anticipating political decision-making relevant to science: a set of beliefs about how and when decisions are produced, what drives them, and what effects they may have on science funding or regulation. These models can be understood as internally constructed filters to help select what is relevant from what is noise or redundancy. They help the system to sort through what is expected and what is unpredicted, what is a relevant signal and what is an irritation (Luhmann, 1991, p 1432).

If we accept that science and politics are guided by distinct logics or communicative codes, the challenge becomes one of reconstructing how each system might selectively pick up signals from the other. We need to understand what sort of perceptual filters are developed and stabilised for the purpose of screening out relevant signals from noise; and how information from the other system might be constructed and connected to the receiving system’s identities and functions. The implication is that we need to turn our attention to how the system of politics ‘models’ the system of science, and how it selectively appropriates and gives meaning to the signals produced by that system.

This segues nicely into the earlier discussion of our first set of theories, and the need for a more sophisticated theory of politics than those provided by prevalent models of research-policy relations. Such a theory would require an account of how the political system makes sense of its environment, and selectively draws on different types of resources to secure legitimacy or support (Boswell, 2009a). A number of theories from public policy can contribute towards such an endeavour. Notably, theories of information-processing offer potential to examine how organisations in the public administration selectively pick up signals from their environment about social problems (e.g., Baumgartner and Jones, 1993). Cohen and colleagues’ (1972) ‘garbage can model’ of policymaking, as taken up by Kingdon (1995 [1984]), offers a neat way of theorising how different ideas or ‘solutions’ are picked up depending on the political and problem streams–again, an idea broadly compatible with the systems theory approach, in that it views ‘ideas’ and ‘politics’ as operating according to different temporalities and logics (Boswell and Rodrigues, 2016).

What are the implications of systems theory for impact? A systems theoretic approach would be wary of the attempt to demonstrate ‘impact’, as it assumes a specious causality between science and politics. Instead, we need to try to adopt the perspective of politics, and make sense of how and why the political system picks up data, methods or techniques from social science. And we can attempt to observe how, from the perspective of social science, political decisions or goals might affect the selection and framing of research questions, and the communication of research findings. But we cannot integrate these observations into a single set of causal mechanisms. Viewed from ‘inside’ of each system, the other remains a ‘black box’: an infinitely complex set of communications and operations which can only be very crudely modelled and selectively responded to.

What this implies is that an impact case study could at best chart how politics appropriated and gave meaning to particular data, methods or techniques. But the ‘underpinning research’ that produced these data or techniques, or academic efforts to promote this research, would derive rather limited credit for such take-up. Far more important would be dynamics internal to the political system, such as the political salience of the issue, or how well the research in question was attuned to dominant political framings of policy problems (Kingdon, 1995 [1984]; Cairney, 2016), or how far research was seen as an authoritative mode of knowledge for guiding decisions (Boswell, 2009b). Moreover, it would remain open how far political take-up reflected a preoccupation with signalling legitimacy, rather than informing policy interventions. After all, if research is valued by politics as a means of substantiating claims or bolstering credibility, then presumably this implies a symbolic rather than instrumental rationale for using research (Boswell, 2009a).

In short, the systems theoretic account guides us towards an interrogation of the political context of knowledge utilisation; but the more we probe the logic of knowledge appropriation in politics, the less we can accredit research. What makes for politically useful knowledge is fundamentally distinct from what makes for good science. Thus any link between high quality science and impact is exposed as contingent. It may well be that politics needs to ‘quality control’ the science it invokes to insure against its invalidation by critics–but this is only as an insurance against critique. And it may want to ensure the robustness of science as a safeguard against making mistakes that would cost political support. But again, this concern with rigour is incidental to the core concerns of politics. Politics is not fundamentally preoccupied with what is true, but with what is relevant to securing power and producing collectively binding decisions.

Conclusion

Current approaches to research impact appear to have been informed by simplistic supply-side models within our first category of ‘knowledge shapes policy’. As we have suggested in this article, such accounts have been widely debunked by theorists of research-policy relations, as well as by many empirical studies of research ‘impact’. And yet the REF and HEFCE models, and much of the literature on knowledge utilisation, continue to remain faithful to this problematic account. Part of the reason for the sustained commitment to these models is that they offer a reassuring narrative to both policy-makers and researchers. Politicians and public servants can demonstrate the rigour and authority of their claims by invoking research, and they can secure legitimacy by signalling that their decisions are well-grounded (Boswell, 2009a), or they can invoke the need for research as a rationale for delaying action (Fuller, 2005). At the same time, researchers can secure additional resources and credit for developing compelling narratives about the impact of their research (Dunlop, 2017). Yet these accounts bely the complexity of research-policy relations and, indeed, of policy processes and policy change (Cohen et al., 1972; Smith and Katikireddi, 2013). If we are to avoid continually reinventing broken wheels, we suggest a new, more theoretically informed approach to thinking about research impact is required.

The existing literature on research impact has already subjected current approaches to assessing, incentivizing and rewarding impact in the UK to extensive critique, and it was not the purpose of this paper to expand on these critiques. Rather, our aim has been to set out four alternative, sophisticated accounts of the relationship between research and policy and to consider what a research impact agenda might look like if it were informed by these other approaches. Such an exercise is necessarily hypothetical and almost impossible to test in an empirical sense, since the UK’s approach to research impact has already been informed by a relatively simple and linear conceptualisation of research-policy relations (Smith and Stewart, 2016). This means there are strong incentives for institutions to ‘play the game’ according to the rules that have been set by providing relatively simple and linear ‘stories’ of research impact, as Meagher and Martin’s (2017) analysis of REF2014 impact case studies for mathematics attests (see also Murphy, 2017 on ‘gaming’ in REF and Watermeyer and Hendgecoe 2016 on ‘impact mercantilsm’). However, as other countries evolve different approaches to research impact, it may become possible to empirically assess both the claims we set out here and the practical implications of such alternative approaches.

The first of the four models we outline offered a subtler ‘enlightenment’ conception of how research can influence policy. It implied that research can lead to ideational adjustments through diffuse and incremental processes, typically influenced by a wide body of research rather than individual findings. This account challenges the notion that researchers or institutions should be rewarded for claims about the impact of individual studies, though potentially supports efforts to encourage knowledge exchange. The second set of theories implied that policy and politics shape knowledge production and use, and were altogether more sceptical of the impact agenda. They suggested that it was naïve to assume that researchers can speak truth to power, implying that researchers should not be rewarded for their supposed impact since policy actors employ research for political, rather than empirical/intellectual, reasons. The third set of theories on co-production implied the need for a far more sophisticated methodology for examining how research and governance are mutually constitutive. They also argued that social science should not necessarily be understood as the ‘solution’ to social problems, since it can itself create such problems. And the fourth approach, which posits that science and politics are autonomous systems, suggested that we can best understand impact through a theory of how politics selectively observes and gives meaning to communications emanating from the system of science. Viewed from this perspective, the impact agenda has been designed to suit the needs of a political, rather than scientific, system and should be treated cautiously by researchers given its potential to divert science from its core task of developing truth claims.

Both the second and fourth accounts suggest that the very idea of trying to incentivize the use of research in policy is flawed. On these accounts, we should be cautious about adopting systems that reward researchers for influencing policy. Such impacts are spurious, in that their apparent influence is down to pre-given interests or independent political dynamics; or they are the result of researchers aligning research questions and approaches to pre-fit political agendas. By rewarding researchers for achieving impact we are adopting an arbitrary incentive system that is at best decoupled from research quality, and at worst, threatens the integrity and independence of social science.

For those more sympathetic to the idea of ‘research impact’, the first and third approaches might offer more hope. Nonetheless, neither approach suggests that the current approach is likely to achieve its intended goals. Indeed, both caution against rewarding individual researchers for ‘achieving’ research impact based on narrow indicators (e.g., citations in policy documents). The enlightenment model suggests that research impact involves subtle, incremental and diffuse ideational adjustments over a long period of time, which are generated by a wide range of research insights rather than specific individual findings. This suggests that a system for rewarding impact should not focus on individual research projects or groups and their linear effects on particular policies. Rather, impact frameworks should reward collaborative endeavours that build incrementally on a wider body of work; that develop longer-term relationships with a range of non-academic audiences (not only policymakers and other ‘elites’); and that may bring about subtle conceptual shifts, rather than clearly identifiable policy changes. This in turn implies the need for more complex research designs and methodologies for charting such influence over a far longer time-frame, and avoid incentives to over-claim credit for particular groups or projects. This perspective coheres with those arguing for a shift away from trying to measure and incentivize research impacts to focus instead on incentivizing and rewarding knowledge exchange processes (e.g., Upton et al., 2014). From this view, Spaapen and van Drooge’s (2011) approach of focusing on ‘productive interactions’ between science and society (which emerged out of an FP7 project called Social Impact Assessment Methods for research and funding instruments-SIAMPI), seems like a more defensible means of assessing research impact. The notion of co-production similarly suggests the need for more in-depth, ethnographic or process-tracing methods for reconstructing the complex relationships between research and policy (as outlined by Baim-Lance and Vindrola-Padros, 2015). Systems for rewarding impact should also be aware of the two-way relationship between research and governance, including the ways in which social science can itself affect the social and political world, imagining and enacting new social problems.

Arguably, the highest impact research is that which serves to re-shape the social world it seeks to describe. This implies that models to promote engagement with knowledge users need to be attentive not just to the complex pathways to research impact, but also to the very real ethical implications of research influence (implications that do not currently appear to be considered in either REF impact case studies or RCUK pathways to impact statements–Smith and Stewart, 2016). Not only can the impact agenda affect the practices of social science, as is widely recognised in social science literature; social science can also instigate new policy problems. Proponents of policy impact should have a care what they wish for.

Data availability

The article does not generate or make use of any datasets.