Posts Tagged ‘“research assessment”’
Research quality: responses from the floor
Questions from the audience, and answers from the panel, relating to Richard "call me Dimbleby" Gedye and the Four Horses of the Research Quality Apocalypse
Ed Pentz (audience): is there a common definition / agreement on what quality or impact actually is?
Jim Pringle: we rarely articulate this. Quality is in the eye of the beholder - subjective. Is the work making a significant contribution to the field, changing it? Is it well constructed, convincing? From the funders' point of view, was it worthwhile?
Hugh Look: ye-es. I'm pessimistic about this. We can come up with words that everyone agrees on but will it really mean anything? We just manufacture more specialist language. (Rick Anderson summarises this on Twitter as "Shared language" doesn't necessarily equal "common understanding." - exactly.)
Alain Peyraube: who decides what is making a significant contribution to the field? It depends on judgements that inherently don't achieve consensus. And it changes over time.
Peter Shepherd: the easiest way to describe quality is that you know it when you see it! It helps create insight, and enable others to take things further forward.
Hugh (again): if peer review and metrics lead to risk-averse decisions, what do we do? Encourage plurality and diversity. Metrics have a chilling, crushing effect on plurality and diversity. We do need to do riskier things, that we can't foresee the outcome of. That is being lost from the system because we are focussing on targets.
Jim: quality is associated with value and worth. We talk about it because of related funding decisions. Challenge: what would be the evidence of worth and value that we should look for? The issues we have discussed are a failure of management to use tools correctly; they don't mean there should be no search for measurable value.
Judy Luther (audience): what's your sense of the future of the journal as a signifier of research quality?
Peter: journals are a service to authors, and a service to readers. On the harvesting side, journals will continue to be an important measure of quality (from the author's point of view) - represented by editors and editorial boards. Within the great mass of information, most of us need something like the journal's personality to trust our research to. From the reader's point of view, the journal as quality is "becoming more shady". The collection (database) or the individual article might become more meaningful than the journal, as a proxy for quality.
Alain: the journal will continue to exist. The problem is that when a paper is published in a journal with a good reputation, the content of the paper has already been discussed, criticised etc (e.g. at conferences - several months prior to publication) so the article is too late to improve science - there is nothing new for specialists in the field; they already know the content, and they don't read it again. (On Twitter, Ed Pentz argues that "research may have moved on but in many fields old articles can have a huge impact"),
Hugh: we might start to see metrics relating to interestingness, relevance, rather than long-term quality. "Perhaps we need to look at temporal variation in the significance and meaning of metrics, to understand in a more sophisticated way what these things are doing for us."
Hazel Woodward (audience): the focus for assigning value is the article / journal, but research funds are allocated to researchers (individuals and institutions). Should we move to allocating research quality points to individuals / institutions, or is it not practical to pursue this?
Hugh: I feel uncomfortable - that this is an ill-thought out idea. It seems to be further atomisation, further subjective judgements, further bureaucracy. I instinctively feel that practitioners should take a fairly aggressive stance against this.
Richard Gedye: do you worry that it's already happening (RAE etc)?
Hugh: I worry about its use. In the end, it rewards the compliant, not the difficult - those who are performing. It fundamentally supports a structure that does not reward those who go against the grain.
Alain: It's already happening (in institutions) - Shanghai rankings etc. Attributing research quality points to researchers would invoke strong reactions.
Jim: if you're a researcher, and you have a body of work that has never been downloaded, never visibly has been read or shown or cited, shouldn't we ask some questions about the value of your research?
Peter: is this Question Time or I'm Sorry I Haven't a Clue? Points mean prizes! I don't in principle have a problem with awarding points, but the problem is the scale of the enterprise sitting behind this. A huge bureaucracy outweighs the benefit of the metric. The cost, in many senses, of playing this game, has become too much.
Ed Pentz (audience): is there a common definition / agreement on what quality or impact actually is?
Jim Pringle: we rarely articulate this. Quality is in the eye of the beholder - subjective. Is the work making a significant contribution to the field, changing it? Is it well constructed, convincing? From the funders' point of view, was it worthwhile?
Hugh Look: ye-es. I'm pessimistic about this. We can come up with words that everyone agrees on but will it really mean anything? We just manufacture more specialist language. (Rick Anderson summarises this on Twitter as "Shared language" doesn't necessarily equal "common understanding." - exactly.)
Alain Peyraube: who decides what is making a significant contribution to the field? It depends on judgements that inherently don't achieve consensus. And it changes over time.
Peter Shepherd: the easiest way to describe quality is that you know it when you see it! It helps create insight, and enable others to take things further forward.
Hugh (again): if peer review and metrics lead to risk-averse decisions, what do we do? Encourage plurality and diversity. Metrics have a chilling, crushing effect on plurality and diversity. We do need to do riskier things, that we can't foresee the outcome of. That is being lost from the system because we are focussing on targets.
Jim: quality is associated with value and worth. We talk about it because of related funding decisions. Challenge: what would be the evidence of worth and value that we should look for? The issues we have discussed are a failure of management to use tools correctly; they don't mean there should be no search for measurable value.
Judy Luther (audience): what's your sense of the future of the journal as a signifier of research quality?
Peter: journals are a service to authors, and a service to readers. On the harvesting side, journals will continue to be an important measure of quality (from the author's point of view) - represented by editors and editorial boards. Within the great mass of information, most of us need something like the journal's personality to trust our research to. From the reader's point of view, the journal as quality is "becoming more shady". The collection (database) or the individual article might become more meaningful than the journal, as a proxy for quality.
Alain: the journal will continue to exist. The problem is that when a paper is published in a journal with a good reputation, the content of the paper has already been discussed, criticised etc (e.g. at conferences - several months prior to publication) so the article is too late to improve science - there is nothing new for specialists in the field; they already know the content, and they don't read it again. (On Twitter, Ed Pentz argues that "research may have moved on but in many fields old articles can have a huge impact"),
Hugh: we might start to see metrics relating to interestingness, relevance, rather than long-term quality. "Perhaps we need to look at temporal variation in the significance and meaning of metrics, to understand in a more sophisticated way what these things are doing for us."
Hazel Woodward (audience): the focus for assigning value is the article / journal, but research funds are allocated to researchers (individuals and institutions). Should we move to allocating research quality points to individuals / institutions, or is it not practical to pursue this?
Hugh: I feel uncomfortable - that this is an ill-thought out idea. It seems to be further atomisation, further subjective judgements, further bureaucracy. I instinctively feel that practitioners should take a fairly aggressive stance against this.
Richard Gedye: do you worry that it's already happening (RAE etc)?
Hugh: I worry about its use. In the end, it rewards the compliant, not the difficult - those who are performing. It fundamentally supports a structure that does not reward those who go against the grain.
Alain: It's already happening (in institutions) - Shanghai rankings etc. Attributing research quality points to researchers would invoke strong reactions.
Jim: if you're a researcher, and you have a body of work that has never been downloaded, never visibly has been read or shown or cited, shouldn't we ask some questions about the value of your research?
Peter: is this Question Time or I'm Sorry I Haven't a Clue? Points mean prizes! I don't in principle have a problem with awarding points, but the problem is the scale of the enterprise sitting behind this. A huge bureaucracy outweighs the benefit of the metric. The cost, in many senses, of playing this game, has become too much.
Richard "call me Dimbleby" Gedye and the Four Horses of the Research Quality Apocalypse
We're not too thin on the ground for this morning's 9am session, which suggests either that the conference dinner was a washout (it wasn't, as witnessed by my slight queasiness and faint headache) or that this morning's session is big draw (it is - a new, Question-Time-style format for UKSG).
Jim Pringle from Thomson Reuters opens the show. We're talking about the value / quality of research, and the ways in which it can be measured. Jim suggests:
Tools are emerging that make use of metadata relationships (funders, co-authors, patents, etc) to more powerfully and accurately measure value. "But metrics are only as good as the people who use them, and can support judgements but should never be the sole ground for judgements."
COUNTER's Peter Shepherd walks us through PIRUS2, a project that is developing a standard for reporting usage at the individual article level. Journal-level metrics may not be representative of individual articles and therefore author, and citation data is not adequate for measuring some fields. The H-Index is author-centred but can be biased towards older researchers. Overall reliance on any one metric is misleading, and distorts author behaviour.
Article-level usage is becoming more relevant because:
Alain Peyraube from CNRS picks up the baton, talking about the European Reference Index for the Humanities project. "We need appropriate tools (for measuring impact) or we are going to lose valuable grants." In the humanities in particular, such a methodology needs to address monographs, book chapters, edited volumes etc as well as journals. The ERIH steering committee took some time to set up the project, identifying which disciplines would be considered (lots of overlap with social sciences), selecting peer review as "the only practicable method of evaluation in basic research", setting up panels for each of 15 disciplines, providing guidelines and soliciting categorised lists of journals. The project suffered from considerable misunderstanding and criticism, especially from the UK scientific community, and particularly around the categorisation of journals. Both the panels and the journal lists have been revised since the project's inception.
Hugh Look takes the stage in a fabulous red blazer (Good Morning Campers!). Trying to attach numbers to things, says Hugh, is a difficult business - numeric targets are always open to abuse, and he points us to the LSE's Michael Power for a widespread analysis of this. What's the benefit of measuring - and who benefits? What are the underlying structures, and what behaviours do they engender? How strong is the link between measurements and real-world impact? We're not currently measuring performance or quality - we're setting up a structure of control, and this risks unproductive use of public money. "The managerial class are the primary beneficiaries of the measurement culture." They manage risk to the institution - but also the risk to themselves, so they use metrics to safeguard their own position / avoid bad PR, and coerce practitioners into compliance by making it part of annual reviews. There is a flight from judgement in the way that metrics are used. Managers cannot do peer review themselves, so they mistrust it.
Hugh acknowledges that measuring things can break "who you know" elitism, but both he and Peter referred to "fetishism" around statistics. Some organisations, e.g. the RAF, are stepping back from using metrics to assess research quality - a "dawning of common sense". Metrics divert attention away from "other things that are more useful to do" - "what aren't we doing?"
(Questions / floor discussion I will post separately, or I'll be verging on LiveSerials' longest posting).
Jim Pringle from Thomson Reuters opens the show. We're talking about the value / quality of research, and the ways in which it can be measured. Jim suggests:
- attention (citations, but also more generally)
- aggregation (researcher -> institution, article -> journal)
- relation (links, related content, metadata - lots of data stored around the institution can be useful in assessing researcher value / impact).
Tools are emerging that make use of metadata relationships (funders, co-authors, patents, etc) to more powerfully and accurately measure value. "But metrics are only as good as the people who use them, and can support judgements but should never be the sole ground for judgements."
COUNTER's Peter Shepherd walks us through PIRUS2, a project that is developing a standard for reporting usage at the individual article level. Journal-level metrics may not be representative of individual articles and therefore author, and citation data is not adequate for measuring some fields. The H-Index is author-centred but can be biased towards older researchers. Overall reliance on any one metric is misleading, and distorts author behaviour.
Article-level usage is becoming more relevant because:
- more journal articles in repositories
- interest from authors and funding agencies
- online usage growing in credibility e.g. PLoS reporting, Knowledge Exchange
- increased practicality - COUNTER, SUSHI
Alain Peyraube from CNRS picks up the baton, talking about the European Reference Index for the Humanities project. "We need appropriate tools (for measuring impact) or we are going to lose valuable grants." In the humanities in particular, such a methodology needs to address monographs, book chapters, edited volumes etc as well as journals. The ERIH steering committee took some time to set up the project, identifying which disciplines would be considered (lots of overlap with social sciences), selecting peer review as "the only practicable method of evaluation in basic research", setting up panels for each of 15 disciplines, providing guidelines and soliciting categorised lists of journals. The project suffered from considerable misunderstanding and criticism, especially from the UK scientific community, and particularly around the categorisation of journals. Both the panels and the journal lists have been revised since the project's inception.
Hugh Look takes the stage in a fabulous red blazer (Good Morning Campers!). Trying to attach numbers to things, says Hugh, is a difficult business - numeric targets are always open to abuse, and he points us to the LSE's Michael Power for a widespread analysis of this. What's the benefit of measuring - and who benefits? What are the underlying structures, and what behaviours do they engender? How strong is the link between measurements and real-world impact? We're not currently measuring performance or quality - we're setting up a structure of control, and this risks unproductive use of public money. "The managerial class are the primary beneficiaries of the measurement culture." They manage risk to the institution - but also the risk to themselves, so they use metrics to safeguard their own position / avoid bad PR, and coerce practitioners into compliance by making it part of annual reviews. There is a flight from judgement in the way that metrics are used. Managers cannot do peer review themselves, so they mistrust it.
Hugh acknowledges that measuring things can break "who you know" elitism, but both he and Peter referred to "fetishism" around statistics. Some organisations, e.g. the RAF, are stepping back from using metrics to assess research quality - a "dawning of common sense". Metrics divert attention away from "other things that are more useful to do" - "what aren't we doing?"
(Questions / floor discussion I will post separately, or I'll be verging on LiveSerials' longest posting).
Posted in New Law Books | 
Tags: "research assessment", citations, impact factor, metrics, Quality, Quality Measures, usage, Usage Factor, value