Posts Tagged ‘impact factor’
UKSG Conference Summary
This is the summary I put together for my SAGE colleagues. I am reposting here in case it is of use. Comments on theme omissions welcome.
The 33rd Annual UKSG Conference was held in Edinburgh a couple of weeks ago, with a varied programme and over 850 attendees. Themes gleaned from the sessions and discussions I attended are summarised below. The very active twitter stream from the conference can be found here. I was amazed at the ability of delegates to listen, assimilate and then tweet or blog all at the same time!
Social Media
An increase in focus on Social Media and new ways of sharing information and research was very evident. There was a much higher engagement and understanding of newer communication channels. Interestingly, it was not younger researchers quoted as spending more time blogging or commenting online but more established researchers who have already secured a reputation and can afford to spend time in this way. They are contributing to knowledge and growth in their field but not just in the tradition of peer-reviewed high impact journals. An interesting development would be a unique researcher identifier such as ORCID which would help tie all researcher output together.
Researchblogging is interesting, its focus is on serious posts about peer reviewed academic research. It has over 1200 blogs and has doubled in the last year. Adam Bly from Researchblogging explained that more collaborative science and the creation of new knowledge has the scientific information industry running to keep up. We’ll be contacting them to ensure posts on Sage content can be linked back to the original research.
Open Access
OA feels like it is slowly gaining traction as a publishing model. The vibe from various sessions was less if than when. Tony Hirst was emphatic on the new openness of communication channels. Why use traditional journal articles to share ideas he asked? If he or his colleagues do decide to publish an article it will be in OA jnls. However, no clear wide-ranging path for institutions to fund OA was apparent yet.
Usage
JISC Collections have secured funding to progress with the JISC usage stats portal or “make it real”. We'll be contacted for our input, the prototype involved data from Elsevier, OUP and Springer. Its aim is to present usage data in user-friendly ways for librarians and include new ways to benchmark usage. There is not clarity on confidentiality issues and was no resolution proposed on the question of the confidential nature of usage data. It’s early days.
Big Deals, Value & Pricing
An interesting session on value had Ted Bergstrom from UCSB explaining that he is securing information about big deal pricing so he can publish information about outliers in the public domain. His reason: “as citizens of the academic community, we are interested in helping librarians to understand the dynamic economic problem that they face and aiding them in negotiating effectively with large publishers. We plan to release a collection of information and analyses that will serve this purpose.” See his Big Deal Contract Project page.
As in previous years the mood seemed to be that the Big Deal bubble must burst, as it is unsustainable for many institutions, but there was no clear way forward proposed still.
Carol Tenopir talked about developing real tools for librarians to demonstrate value of their collections. She pointed us to the ARL website for more information.
Jill emery from Texas talked about patron-driven ebook and journal article acquisition stating 'the age of the article is here'. She explained they need “aggregated article access”. She wants publishers to listen as they have to purchase “Just in time not just in case'.
Quality Metrics
Frustration with the reliance on the Impact Factor and the fact it ranks journals and not articles was apparent. There is a desire to produce new ranking metrics. the Australian Research Assessment is no longer using the Impact Factor we were told.
Pete Binfield (PLOS) ran a session on how they have introduced article-level metrics - it’s worth a look if you haven’t seen. They recognise they are very much at the beginning but are very keen to do whatever they can to help users decide which content is highly valued by the community. Also, these metrics not just about evaluation but to help users filter and discover articles of value.
Pete talked about how they have work to do on working out how to measure “influence”. It’s important to demonstrate influence beyond the scientific community. This ties in with our work to show the value of social science research. How do we ensure the research we publish is credited appropriately when it influences Government policy for instance?
As Hannah Whaley says on her blog: The discussion around these issues is healthy, as is the growing volume with which librarians and researchers are willing to speak them out loud. However these key themes are notable for representing problems, not solutions. It is clear that licensing models, researcher metrics, electronic and open access still have some way to evolve to meet the growing needs and expectations of the community.
Posted in New Law Books | 
Tags: article-level metrics, big deal, blog, impact factor, open access, research, sage, social networking, summary, value
Hannah Whaley’s UKSG summary
Hannah Whaley is an Assistant Director in the University of Dundee’s Library and Learning Centre, with responsibility for Research and Systems. She specialises in system design, service development and innovation within HE teaching and research. Hannah recently wrote a great blog posting identifying key themes at this year's UKSG conference, and she's kindly agreed that we can syndicate her posting here. Read on for a snapshot of the conference from Hannah's point of view - does it tally with yours? More snapshots coming soon!
(Syndicated with permission from http://www.hannahwhaley.com/2010/04/18/uksg-main-themes/)
The 33rd Annual UKSG Conference was in Edinburgh this week, with a varied programme and over 850 attendees. A number of themes started to recur through the sessions and discussions, as summarised:
- Big deal bubble must burst, as it is unsustainable for many institutions
- We must move further towards open access, but it is not yet clear how
- Journal impact factor isn’t good enough anymore, we need to review the commentary and produce new ranking factors
- Linked information is nearly here, allowing informal and pre-publish conversations to be viewed and measured in a structured way on the web
- The age of the article is here, meaning metrics, usage and discoverability will increasingly be at article level rather than the ‘journal container’
- Just-in-time must replace just-in-case, as no one can maintain a full array of items that may only occasionally be required
The discussion around these issues is healthy, as is the growing volume with which librarians and researchers are willing to speak them out loud. However these key themes are notable for representing problems, not solutions. It is clear that licensing models, researcher metrics, electronic and open access still have some way to evolve to meet the growing needs and expectations of the community.
(Syndicated with permission from http://www.hannahwhaley.com/2010/04/18/uksg-main-themes/)
Richard "call me Dimbleby" Gedye and the Four Horses of the Research Quality Apocalypse
We're not too thin on the ground for this morning's 9am session, which suggests either that the conference dinner was a washout (it wasn't, as witnessed by my slight queasiness and faint headache) or that this morning's session is big draw (it is - a new, Question-Time-style format for UKSG).
Jim Pringle from Thomson Reuters opens the show. We're talking about the value / quality of research, and the ways in which it can be measured. Jim suggests:
Tools are emerging that make use of metadata relationships (funders, co-authors, patents, etc) to more powerfully and accurately measure value. "But metrics are only as good as the people who use them, and can support judgements but should never be the sole ground for judgements."
COUNTER's Peter Shepherd walks us through PIRUS2, a project that is developing a standard for reporting usage at the individual article level. Journal-level metrics may not be representative of individual articles and therefore author, and citation data is not adequate for measuring some fields. The H-Index is author-centred but can be biased towards older researchers. Overall reliance on any one metric is misleading, and distorts author behaviour.
Article-level usage is becoming more relevant because:
Alain Peyraube from CNRS picks up the baton, talking about the European Reference Index for the Humanities project. "We need appropriate tools (for measuring impact) or we are going to lose valuable grants." In the humanities in particular, such a methodology needs to address monographs, book chapters, edited volumes etc as well as journals. The ERIH steering committee took some time to set up the project, identifying which disciplines would be considered (lots of overlap with social sciences), selecting peer review as "the only practicable method of evaluation in basic research", setting up panels for each of 15 disciplines, providing guidelines and soliciting categorised lists of journals. The project suffered from considerable misunderstanding and criticism, especially from the UK scientific community, and particularly around the categorisation of journals. Both the panels and the journal lists have been revised since the project's inception.
Hugh Look takes the stage in a fabulous red blazer (Good Morning Campers!). Trying to attach numbers to things, says Hugh, is a difficult business - numeric targets are always open to abuse, and he points us to the LSE's Michael Power for a widespread analysis of this. What's the benefit of measuring - and who benefits? What are the underlying structures, and what behaviours do they engender? How strong is the link between measurements and real-world impact? We're not currently measuring performance or quality - we're setting up a structure of control, and this risks unproductive use of public money. "The managerial class are the primary beneficiaries of the measurement culture." They manage risk to the institution - but also the risk to themselves, so they use metrics to safeguard their own position / avoid bad PR, and coerce practitioners into compliance by making it part of annual reviews. There is a flight from judgement in the way that metrics are used. Managers cannot do peer review themselves, so they mistrust it.
Hugh acknowledges that measuring things can break "who you know" elitism, but both he and Peter referred to "fetishism" around statistics. Some organisations, e.g. the RAF, are stepping back from using metrics to assess research quality - a "dawning of common sense". Metrics divert attention away from "other things that are more useful to do" - "what aren't we doing?"
(Questions / floor discussion I will post separately, or I'll be verging on LiveSerials' longest posting).
Jim Pringle from Thomson Reuters opens the show. We're talking about the value / quality of research, and the ways in which it can be measured. Jim suggests:
- attention (citations, but also more generally)
- aggregation (researcher -> institution, article -> journal)
- relation (links, related content, metadata - lots of data stored around the institution can be useful in assessing researcher value / impact).
Tools are emerging that make use of metadata relationships (funders, co-authors, patents, etc) to more powerfully and accurately measure value. "But metrics are only as good as the people who use them, and can support judgements but should never be the sole ground for judgements."
COUNTER's Peter Shepherd walks us through PIRUS2, a project that is developing a standard for reporting usage at the individual article level. Journal-level metrics may not be representative of individual articles and therefore author, and citation data is not adequate for measuring some fields. The H-Index is author-centred but can be biased towards older researchers. Overall reliance on any one metric is misleading, and distorts author behaviour.
Article-level usage is becoming more relevant because:
- more journal articles in repositories
- interest from authors and funding agencies
- online usage growing in credibility e.g. PLoS reporting, Knowledge Exchange
- increased practicality - COUNTER, SUSHI
Alain Peyraube from CNRS picks up the baton, talking about the European Reference Index for the Humanities project. "We need appropriate tools (for measuring impact) or we are going to lose valuable grants." In the humanities in particular, such a methodology needs to address monographs, book chapters, edited volumes etc as well as journals. The ERIH steering committee took some time to set up the project, identifying which disciplines would be considered (lots of overlap with social sciences), selecting peer review as "the only practicable method of evaluation in basic research", setting up panels for each of 15 disciplines, providing guidelines and soliciting categorised lists of journals. The project suffered from considerable misunderstanding and criticism, especially from the UK scientific community, and particularly around the categorisation of journals. Both the panels and the journal lists have been revised since the project's inception.
Hugh Look takes the stage in a fabulous red blazer (Good Morning Campers!). Trying to attach numbers to things, says Hugh, is a difficult business - numeric targets are always open to abuse, and he points us to the LSE's Michael Power for a widespread analysis of this. What's the benefit of measuring - and who benefits? What are the underlying structures, and what behaviours do they engender? How strong is the link between measurements and real-world impact? We're not currently measuring performance or quality - we're setting up a structure of control, and this risks unproductive use of public money. "The managerial class are the primary beneficiaries of the measurement culture." They manage risk to the institution - but also the risk to themselves, so they use metrics to safeguard their own position / avoid bad PR, and coerce practitioners into compliance by making it part of annual reviews. There is a flight from judgement in the way that metrics are used. Managers cannot do peer review themselves, so they mistrust it.
Hugh acknowledges that measuring things can break "who you know" elitism, but both he and Peter referred to "fetishism" around statistics. Some organisations, e.g. the RAF, are stepping back from using metrics to assess research quality - a "dawning of common sense". Metrics divert attention away from "other things that are more useful to do" - "what aren't we doing?"
(Questions / floor discussion I will post separately, or I'll be verging on LiveSerials' longest posting).
Posted in New Law Books |
Tags: "research assessment", citations, impact factor, metrics, Quality, Quality Measures, usage, Usage Factor, value