This background paper looks at a wide variety of methods of measuring the impact of research. Each method has its own characteristics and advantages. No one method is complete in itself and does not offer unambiguous or certain results. Measuring the impact of research is necessary and can be useful but it is important to use the results of such evaluations with care.
Current economic conditions and the increasing competition for government funding are leading to an increased focus on the impact of research. Other factors contributing to this interest include the development of ERA which measures the academic excellence of research and the view that impact measures would complement this assessment; a general concern to improve the operation of the national innovation system; and the need to demonstrate to the public that research funds are well-spent.
Measuring the impact of research is difficult because not all impacts are direct and some can be negative or result from the identification of problems that require a non-research response. The time between the performance of research and when its benefits become apparent can be significant, unpredictable and differ for different kinds of research.
The likelihood of research having impact depends not just upon the potential of the research but also on the willingness and ability of players in the wider innovation system to make use of the research; and any research does not exist in isolation but draws on the work of other researchers. In assessing impact it is necessary to acknowledge that research aiming to achieve impact will often have a high risk of failure and that there can be different perspectives about whether a particular impact is positive or not.
Attempts to measure impact, especially if these are ongoing, can distort behaviours such that they might diminish the probability of research reaching its maximum impact. A rigorous assessment of research impact has also to develop appropriate counterfactuals and consider opportunity costs.
Research can have impact through many routes and in different ways. These range from building national capability through advancing knowledge and supporting university teaching, to producing a direct financial return to the institution performing the research and having major economic impact through increases in productivity, employment, competitiveness and business formation. Research can also contribute to national wellbeing through its social impacts and by improving environmental management and sustainability. There are also many intangible benefits of research which are nevertheless real and of value – including on national reputation and attractiveness as a place to learn, work and invest.
There is a wide range of methods that it is possible to use to evaluate research impact. They can operate at the level of individual projects or programs, institutions and nations. Each method has its own characteristics and advantages. While different methods can appeal to different target groups, none is complete in itself and none offers unambiguous or certain results. Studies of the same project or program at different times or across different time spans can produce widely varying results, reflecting the uncertainty of research and the way in which the value of research outputs can change, depending on the context within which they exist – including subsequent advances in research.
Measuring the impact of research is necessary and can be useful but it is important to use the results of such evaluations with care, recognising their fragility. In particular, it is important to assess impact in terms of the impacts the research aimed to achieve, not across all impacts which are possible.