'You only assess what you care about': A new report looks at how research is assessed in Australia

Research plays a pivotal role in society. Through research, we gain new understandings, test theories and make discoveries.

It also has a huge economic value. In 2021, the CSIRO found every A$1 of research and development investment in Australia creates an average of $3.50 in economy-wide benefits.

But how do we know if individual research projects being conducted in Australia are good quality? How is research recognized? The key way this happens is through "research assessment."

What is research assessment?

Research assessment is not a centralized or necessarily formal process. It can involve various processes and measures to evaluate the performance of individual researchers and research institutions. This includes assessing the quality, excellence and impact of various outputs.

Research assessment can be qualitative or quantitative. It can include publications in journals and the number of people who cite the research, gaining grants to do further research, commercialization, media engagement and impact on decision-making or public policy, prizes and invitations to speak at conferences.

If research assessment is working fairly and effectively, it should achieve several things. This includes: helping to develop researchers' careers, making sure innovative research does not get avoided in favor of short-term gains and helping funders and the community have confidence research is providing value for money and adding to the public good.

Our project

Our new project aimed to provide a better understanding of how research assessment affects research in Australia.

In a report released today, we surveyed more than 1,000 Australian researchers and more than 50 research organizations.

This included universities, research institutes, industry bodies, government and not-for-profit organizations. The majority of researchers (74%) were in academic roles. Across those research sectors, we also conducted 11 roundtables involving around 120 people and 25 intensive interviews to understand the issues.

This work was commissioned by Chief Scientist Cathy Foley and conducted by the Australian Council of Learned Academies (involving the academies of science, medical science, engineering and technological sciences, social sciences and humanities).

It also comes as the Universities Accord review examines how research is funded and approached within higher education.

What we found

We found some difficulties with the current approach to research assessment.

We heard there is a tendency by some researchers to "play it safe" in terms of doing research they believe will score well. We also heard how the assessment process can unintentionally exclude or devalue particular forms of knowledge, particularly in the humanities and the social sciences, where outputs can be less easily quantified or less immediately seen.

As one interviewee said, "What is assessed and how it is assessed are an indication of what the organization values. You only assess what you care about. Values and culture drive assessment."

Our roundtables told us senior staff and supervisors are often seen to reinforce the culture of "publish or perish," with the number of articles being valued more highly than the quality.

We heard early and mid-career researchers and people from underrepresented backgrounds can have difficulties trying to "play the game" to advance their careers. For example, early-career researchers are often expected to produce work that benefits their larger team, at a cost to their own capacity for promotion.

As one interviewee noted, "Metrics are essential for defining value and comparative difference, but Australia requires a modern and fair framework for assessing our current and next generation of researchers."

Survey results

Our survey found a high level of dissatisfaction with the state of research assessment. This included:

• 73% of respondents agreed assessment processes are not consistently or equitably applied across disciplines, in particular between the humanities and the sciences
• 67% said there are not enough opportunities to provide input into research assessment practices
• 70% said assessments took up unreasonable time and effort.

The way forward

In our survey, we asked "What is one specific change you would recommend to improve current research assessment processes?"

Respondents wanted to see a shift towards quality over quantity. This means not just a focus on publishing as many papers as possible, but supporting research that may take longer for its value and benefits to emerge.

They wanted interdisciplinary research to be promoted and rewarded, because many of the complex problems of our world—from climate change to domestic violence to housing affordability—require multiple disciplines to be involved in finding solutions. In the same vein, they also wanted collaboration and team work to be rewarded more clearly and transparently.

They wanted less bias towards STEM (science, technology, engineering and math) research and more promotion of diversity and of early-career researchers. This included better understanding of their personal and cultural situation, more focused career development and better managed teamwork.

To achieve all of this, and more, we will also need to understand that no single measure can assess all research or researchers. So, several tools will be needed, including quantitative indicators as well as qualitative measures and peer review.

This article is republished from The Conversation under a Creative Commons license. Read the original article.