Responsible metrics

There are various philosophical discussions about the merits of using citation scores and metrics as measures of research quality, and it is important to note different cultures of citation, authorship and publication rates in different disciplines.  There is a wide range of activity in the area of research metrics and impact measurement to ensure that measures are used responsibly, and that stakeholders experience a level playing field, notably: 

• The University is a signatory of the San Francisco Declaration on Research Assessment (DORA) and has issued a Commitment to Responsible Research Assessment.
• 'The Metric Tide' report (2015) informed the use of metrics for REF 2021.  Its follow-on report, Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment was published in December 2022.  
• The Leiden Manifesto's 10 principles to guide research evaluation was published as a comment in Nature.
• The CoARA (Coalition for Advancing Research Assessment), initiative began January 2022 and builds on DORA, the Leiden Manifesto and the Metric Tide.  Institutions are invited to sign the CoARA Agreement on Reforming Research Assessment.  UKRI is a signatory.

Three key themes in responsible metrics are:

• The need to avoid and challenge use of the Journal Impact Factor (JIF) as a proxy measure of quality for individual research articles
• The importance of only comparing like with like.  For example, do not compare researchers at different career stages, or in different disciplines.  The use of normalised indicators goes some way to addressing this issue.
• Advocating the use of a range of impact measures in research assessment, and the importance of qualitative judgement, including peer-review. 

Responsible research assessment for researchers

DORA has four recommendations specifically for researchers:

15. When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than publication metrics.
16. Wherever appropriate, cite primary literature in which observations are first reported rather than reviews in order to give credit where credit is due.
17. Use a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research outputs.
18. Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote and teach best practice that focuses on the value and influence of specific research outputs.

Next generation metrics

The Global Research Council’s 2021 Conference Report “Responsible research assessment” acknowledged that “research assessment shapes research culture”, impacting on how researchers undertake and disseminate their research.  Ideally research assessment should encourage team science, transdisciplinary science, collaboration, open research, reproducibility or transparency, EDI and enable greater mobility between industry and academia.  Next generation metrics attempt to address these problems, and we outline some of the initiatives below.

• Before a publication appears with author names attached, a huge amount of work goes into the associated research project.  Contributions will have been made by many people whose name won’t appear on the paper as an author.  Initiatives such as CRediT (Contributor Roles Taxonomy) are being explored, to give recognition to people in roles other than principal investigator and lead researchers. 
• Following on from the Metric Tide (2015), Wilsdon et al. published the 2017 report Next-generation metrics: responsible metrics and evaluation for open science.  The report aims to guide further work by including 12 targeted recommendations along the themes of the European Open Science Agenda - developing research infrastructures, fostering and removing barriers to Open Research, and embedding Open Research in society.  To learn more about Open Research practices, please consult our Open Research page and Open Research Canvas course.
• In 2020 CESAER, the association of European Science and Technology Universities, published a white paper on Next Generation Metrics in the light of greater openness in research, with the aim of moving away from a culture of competition- and reputation-based assessment towards a culture of quality, trust and risk-taking.
• The Hong Kong Principles were devised to reward and recognise scholars for behaviour that contributes to trustworthy research.
• UKRI’s Future Research Assessment Programme (FRAP) seeks to understand what a healthy, thriving research system looks like and how an assessment model can best form its foundation. It began reporting in 2023 on strands including: 
  • • evaluation of REF 2021
    • understanding international research assessment practice
    • investigating possible evaluation models that encourage excellent research and impact, support a positive research culture, and reduce the administrative burden on the HE sector.

Further help

Library Services offers online training on responsible metrics via our Influential Researcher Canvas Course

For one-to-one appointments and bespoke workshops, contact the Research Skills Team in Library Services.