Evaluating Societal Impact of Research

Wednesday 29 August

By Anne Paavolainen, PhD student at the University of Turku, School of Economics

The problem, as described by Lutz Bornmann in a 2012 paper, begins from defining ”societal impact of research”. This idea, as a single concept, has not been agreed upon.

“Third stream activities”, ”societal benefits”, ”societal quality”, ”usefulness”, meeting ”public values”, “knowledge transfer” and “societal relevance” are all concepts currently used to describe the societal impact of research. There is no accepted framework with adequate datasets enabling calculations of metric values, comparable to h-index or journal impact factor, when considering the measurement of this influence.

A 2011 study noted, ”In many studies, the societal impact of research has rather been postulated than demonstrated”. Some researchers consider that social, cultural, political and organisational dimensions seem to be missing from the picture, and most research, if noting societal impact, states economic impact.

As the allocation of research funds is increasingly oriented towards research with societal impact, and not based on citations and prizes only, national evaluations, researchers and other external funding agencies are to demonstrating more interest towards societal impact.

A broad definition of societal impact seems to be preferred. The notion is reinforced in an interview with Senior Ministerial Adviser Erja Heikkinen, Head of Science Policy Team at the Finnish Ministry of Education and Culture:

“The discussion of what is societal impact of research is a topic emerging more frequently in a variety of contexts, especially from the viewpoint of whether it is something that universities and other higher research and education institutions should be doing as part of their normal activities, or as a separate or complementary function adding value to research done”, says Dr. Heikkinen. “I’d see, that when research is being conducted, it has different impacts and influences. The research has its own value as it is, as a foundation for civilisation and education, building capacities for operating in the current world.”

Dr. Heikkinen continues on a notion of commercialisation and patents: “For people to actively study, research, challenge themselves and their research and reference groups to pursue things and new notions of the world and realities is valuable. There are also discussions on new products and services that could be developed based on research.”

“It’s also important to understand that all new ideas do not necessarily form an opportunity for commercialisation. To mention an example, the treatment practices keep developing and new innovations are formed. Still, not all new innovations need or can be a new product or service, but this flow between research and practice needs to be kept going. To plaster a hand slightly differently than before can help in the healing process, but does not necessarily require patenting.”

Traditional reporting can, however, miss such an innovation and leave such a new practice unnoticeable (from the viewpoint of impact metrics).

When asked on what aspects of societal impacts of research and reporting of it could be developed, Dr. Heikkinen replies: “What I’d like to hear more is building up involvement and inclusion. Not to understand what is happening in the world and outside of one’s own experiences contributes to isolation and the experience of exclusion. These aspects are at the moment not discussed enough – and would be good to bring visible in evaluating the impact of research and activities of higher education organisations.”

When it comes to reporting and data collected, it’s noticed (in a 2011 study from Rymer, amongst others), that societal impact can be seen as harder to measure than scientific impact, due to lack of datasets to be used across different institutions and disciplines.

“The statistics do not give enough details to fulfil new expectations of financing organisations“, says Dr. Heikkinen, “particularly with regard to the data and indicators that are being collected to represent societal impact. To correct or change that will not be simple. Changes in data collections and reporting compiled at the higher education and research institutions are somewhat international, and there is an interest to keep collecting the same data, to provide data series and comparability.”

“It’s also worthy to note, that volumes of reportable events, such as filed new patents (or other events) can be rather small. If we bring together monetary rewards or funding, link these together with the reportable events, it can result in odd decisions. To continue the example, say, patents in a faculty rise from one to two. Sure, the number of events, here patents, has doubled, but it’s still a small amount.”

According to Dr. Heikkinen, discussions to bring in new types of metrics have been taken, but the availability has been an issue hindering such initiatives. Complementary data, complementing the picture of the societal impact of research would be welcome, but the initiative would be preferred to come from the research institutions and researchers.

During the interview, Dr. Heikkinen showed interest towards altmetrics and quantitative metrics easing comparisons between different researchers, fields of study as well as perhaps, guiding funds.

Discussions on how and when numerical factors on impact can be utilised would be most welcome, to avoid what Hicks and Wouters (2015) call “Impact-factor obsession” and restrain from measuring only what is easy and available.

On the other hand, to set up a completely new measuring system would be burdening to the research institutions, and would not match to the reporting systems elsewhere, making comparisons difficult. Altmetrics could support the traditional reporting and metrics. As Dr. Heikkinen noted, “…there are still the same possibilities and problems as has been with any other channel. But for example, to raise public awareness of research and researchers via social media and other platforms, is a welcomed phenomenon”.

As with any other reporting linked to funding, there is a conflict of interests to provide exactly what is measured, and not the phenomena behind the metrics. As Dr Heikkinen sums up: “…to mention an example, it would not be sustainable to see researchers with great statistics to be hired just before numbers and data for reporting is collected and see the same people to be let go as soon as reporting is completed.”

The interview of Senior Ministerial Adviser Erja Heikkinen, Head of Science Policy Team at the Finnish Ministry of Education and Culture referred to in this blog took place in Helsinki 8th of August 2018.

Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, 13(8), 673–676. Butler, L. (2007) ‘Assessing university research: a plea for a balanced approach’, Science and Public Policy, 34/8: 565–574.

Godin B, Dore C (2005) Measuring the Impacts of Science; Beyond the Economic Dimension, Urbanisation INRS, Culture et Société. Helsinki, Finland: Helsinki Institute for Science and Technology Studies. http:// www.csiic.ca/PDF/Godin_Dore_Impacts.pdf

Niederkrotenthaler T, Dorner TE, Maier M (2011) Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health 11: 588

Hanney S, Packwood T, Buxton M (2000) Evaluating the benefits from health research and development centres: a categorization, a model and examples of application. Evaluation 6: 137–160

Hicks, D. & Wouters, P. (2015). The Leiden Manifesto for research metrics. Nature, 520(7548), 429-431. https://www.nature.com/news/bibliometrics-the-leiden-manifesto-forresearch-metrics-1.17351

Rymer L (2011) Measuring the Impact of Research—The Context for Metric Development. Turner, Australia: The Group of Eight