Posts Tagged ‘value’
UKSG Conference Summary
This is the summary I put together for my SAGE colleagues. I am reposting here in case it is of use. Comments on theme omissions welcome.
The 33rd Annual UKSG Conference was held in Edinburgh a couple of weeks ago, with a varied programme and over 850 attendees. Themes gleaned from the sessions and discussions I attended are summarised below. The very active twitter stream from the conference can be found here. I was amazed at the ability of delegates to listen, assimilate and then tweet or blog all at the same time!
Social Media
An increase in focus on Social Media and new ways of sharing information and research was very evident. There was a much higher engagement and understanding of newer communication channels. Interestingly, it was not younger researchers quoted as spending more time blogging or commenting online but more established researchers who have already secured a reputation and can afford to spend time in this way. They are contributing to knowledge and growth in their field but not just in the tradition of peer-reviewed high impact journals. An interesting development would be a unique researcher identifier such as ORCID which would help tie all researcher output together.
Researchblogging is interesting, its focus is on serious posts about peer reviewed academic research. It has over 1200 blogs and has doubled in the last year. Adam Bly from Researchblogging explained that more collaborative science and the creation of new knowledge has the scientific information industry running to keep up. We’ll be contacting them to ensure posts on Sage content can be linked back to the original research.
Open Access
OA feels like it is slowly gaining traction as a publishing model. The vibe from various sessions was less if than when. Tony Hirst was emphatic on the new openness of communication channels. Why use traditional journal articles to share ideas he asked? If he or his colleagues do decide to publish an article it will be in OA jnls. However, no clear wide-ranging path for institutions to fund OA was apparent yet.
Usage
JISC Collections have secured funding to progress with the JISC usage stats portal or “make it real”. We'll be contacted for our input, the prototype involved data from Elsevier, OUP and Springer. Its aim is to present usage data in user-friendly ways for librarians and include new ways to benchmark usage. There is not clarity on confidentiality issues and was no resolution proposed on the question of the confidential nature of usage data. It’s early days.
Big Deals, Value & Pricing
An interesting session on value had Ted Bergstrom from UCSB explaining that he is securing information about big deal pricing so he can publish information about outliers in the public domain. His reason: “as citizens of the academic community, we are interested in helping librarians to understand the dynamic economic problem that they face and aiding them in negotiating effectively with large publishers. We plan to release a collection of information and analyses that will serve this purpose.” See his Big Deal Contract Project page.
As in previous years the mood seemed to be that the Big Deal bubble must burst, as it is unsustainable for many institutions, but there was no clear way forward proposed still.
Carol Tenopir talked about developing real tools for librarians to demonstrate value of their collections. She pointed us to the ARL website for more information.
Jill emery from Texas talked about patron-driven ebook and journal article acquisition stating 'the age of the article is here'. She explained they need “aggregated article access”. She wants publishers to listen as they have to purchase “Just in time not just in case'.
Quality Metrics
Frustration with the reliance on the Impact Factor and the fact it ranks journals and not articles was apparent. There is a desire to produce new ranking metrics. the Australian Research Assessment is no longer using the Impact Factor we were told.
Pete Binfield (PLOS) ran a session on how they have introduced article-level metrics - it’s worth a look if you haven’t seen. They recognise they are very much at the beginning but are very keen to do whatever they can to help users decide which content is highly valued by the community. Also, these metrics not just about evaluation but to help users filter and discover articles of value.
Pete talked about how they have work to do on working out how to measure “influence”. It’s important to demonstrate influence beyond the scientific community. This ties in with our work to show the value of social science research. How do we ensure the research we publish is credited appropriately when it influences Government policy for instance?
As Hannah Whaley says on her blog: The discussion around these issues is healthy, as is the growing volume with which librarians and researchers are willing to speak them out loud. However these key themes are notable for representing problems, not solutions. It is clear that licensing models, researcher metrics, electronic and open access still have some way to evolve to meet the growing needs and expectations of the community.
Posted in New Law Books | 
Tags: article-level metrics, big deal, blog, impact factor, open access, research, sage, social networking, summary, value
Richard "call me Dimbleby" Gedye and the Four Horses of the Research Quality Apocalypse
We're not too thin on the ground for this morning's 9am session, which suggests either that the conference dinner was a washout (it wasn't, as witnessed by my slight queasiness and faint headache) or that this morning's session is big draw (it is - a new, Question-Time-style format for UKSG).
Jim Pringle from Thomson Reuters opens the show. We're talking about the value / quality of research, and the ways in which it can be measured. Jim suggests:
Tools are emerging that make use of metadata relationships (funders, co-authors, patents, etc) to more powerfully and accurately measure value. "But metrics are only as good as the people who use them, and can support judgements but should never be the sole ground for judgements."
COUNTER's Peter Shepherd walks us through PIRUS2, a project that is developing a standard for reporting usage at the individual article level. Journal-level metrics may not be representative of individual articles and therefore author, and citation data is not adequate for measuring some fields. The H-Index is author-centred but can be biased towards older researchers. Overall reliance on any one metric is misleading, and distorts author behaviour.
Article-level usage is becoming more relevant because:
Alain Peyraube from CNRS picks up the baton, talking about the European Reference Index for the Humanities project. "We need appropriate tools (for measuring impact) or we are going to lose valuable grants." In the humanities in particular, such a methodology needs to address monographs, book chapters, edited volumes etc as well as journals. The ERIH steering committee took some time to set up the project, identifying which disciplines would be considered (lots of overlap with social sciences), selecting peer review as "the only practicable method of evaluation in basic research", setting up panels for each of 15 disciplines, providing guidelines and soliciting categorised lists of journals. The project suffered from considerable misunderstanding and criticism, especially from the UK scientific community, and particularly around the categorisation of journals. Both the panels and the journal lists have been revised since the project's inception.
Hugh Look takes the stage in a fabulous red blazer (Good Morning Campers!). Trying to attach numbers to things, says Hugh, is a difficult business - numeric targets are always open to abuse, and he points us to the LSE's Michael Power for a widespread analysis of this. What's the benefit of measuring - and who benefits? What are the underlying structures, and what behaviours do they engender? How strong is the link between measurements and real-world impact? We're not currently measuring performance or quality - we're setting up a structure of control, and this risks unproductive use of public money. "The managerial class are the primary beneficiaries of the measurement culture." They manage risk to the institution - but also the risk to themselves, so they use metrics to safeguard their own position / avoid bad PR, and coerce practitioners into compliance by making it part of annual reviews. There is a flight from judgement in the way that metrics are used. Managers cannot do peer review themselves, so they mistrust it.
Hugh acknowledges that measuring things can break "who you know" elitism, but both he and Peter referred to "fetishism" around statistics. Some organisations, e.g. the RAF, are stepping back from using metrics to assess research quality - a "dawning of common sense". Metrics divert attention away from "other things that are more useful to do" - "what aren't we doing?"
(Questions / floor discussion I will post separately, or I'll be verging on LiveSerials' longest posting).
Jim Pringle from Thomson Reuters opens the show. We're talking about the value / quality of research, and the ways in which it can be measured. Jim suggests:
- attention (citations, but also more generally)
- aggregation (researcher -> institution, article -> journal)
- relation (links, related content, metadata - lots of data stored around the institution can be useful in assessing researcher value / impact).
Tools are emerging that make use of metadata relationships (funders, co-authors, patents, etc) to more powerfully and accurately measure value. "But metrics are only as good as the people who use them, and can support judgements but should never be the sole ground for judgements."
COUNTER's Peter Shepherd walks us through PIRUS2, a project that is developing a standard for reporting usage at the individual article level. Journal-level metrics may not be representative of individual articles and therefore author, and citation data is not adequate for measuring some fields. The H-Index is author-centred but can be biased towards older researchers. Overall reliance on any one metric is misleading, and distorts author behaviour.
Article-level usage is becoming more relevant because:
- more journal articles in repositories
- interest from authors and funding agencies
- online usage growing in credibility e.g. PLoS reporting, Knowledge Exchange
- increased practicality - COUNTER, SUSHI
Alain Peyraube from CNRS picks up the baton, talking about the European Reference Index for the Humanities project. "We need appropriate tools (for measuring impact) or we are going to lose valuable grants." In the humanities in particular, such a methodology needs to address monographs, book chapters, edited volumes etc as well as journals. The ERIH steering committee took some time to set up the project, identifying which disciplines would be considered (lots of overlap with social sciences), selecting peer review as "the only practicable method of evaluation in basic research", setting up panels for each of 15 disciplines, providing guidelines and soliciting categorised lists of journals. The project suffered from considerable misunderstanding and criticism, especially from the UK scientific community, and particularly around the categorisation of journals. Both the panels and the journal lists have been revised since the project's inception.
Hugh Look takes the stage in a fabulous red blazer (Good Morning Campers!). Trying to attach numbers to things, says Hugh, is a difficult business - numeric targets are always open to abuse, and he points us to the LSE's Michael Power for a widespread analysis of this. What's the benefit of measuring - and who benefits? What are the underlying structures, and what behaviours do they engender? How strong is the link between measurements and real-world impact? We're not currently measuring performance or quality - we're setting up a structure of control, and this risks unproductive use of public money. "The managerial class are the primary beneficiaries of the measurement culture." They manage risk to the institution - but also the risk to themselves, so they use metrics to safeguard their own position / avoid bad PR, and coerce practitioners into compliance by making it part of annual reviews. There is a flight from judgement in the way that metrics are used. Managers cannot do peer review themselves, so they mistrust it.
Hugh acknowledges that measuring things can break "who you know" elitism, but both he and Peter referred to "fetishism" around statistics. Some organisations, e.g. the RAF, are stepping back from using metrics to assess research quality - a "dawning of common sense". Metrics divert attention away from "other things that are more useful to do" - "what aren't we doing?"
(Questions / floor discussion I will post separately, or I'll be verging on LiveSerials' longest posting).
Posted in New Law Books |
Tags: "research assessment", citations, impact factor, metrics, Quality, Quality Measures, usage, Usage Factor, value
Plenary session 2: Economics of Scholarly Information
Ted Bergstrom
Big Deals and the Terrible Fix
Ted started by describing librarians' “shopping problems"
The first big deals entailed a publisher (roughly speaking) working out the library's current spend, multiplying by a factor, and then offering e-access to all content in addition to the print for existing subscriptions.
How effective was this? Very, since the publisher knows that the library is willing to pay at least same as they did for their paper subscriptions, and giving them additional content costs the publisher nothing. Bundling works well from a seller's perspective: demand for bundles varies less than individual titles, and deters entry to the market by making it hard to add new competing journals. And of course, by the time the deal expires, the faculty are addicted to online access so the library has to negotiate a new deal.
Ted then gave the audience an economic prescription: namely, if you want to allocate resources efficiently, use a price system where users pay for what they get out of their own money, and therefore economise, ie a pay per view model. If informed users spend their own money on downloads, demand becomes more price elastic, monopolists are forced to cut prices, and authors prefer to publish in reasonably priced journals in order to get more readers.
Does it work for academics? Ask yourself: who else is better placed to decide value of article access?
Currently there's a big discrepancy between the average cost of a non-profit and for-profit article. The competition prediction suggests that if users pay, it would drive down prices to just over cost-price. There could be limited role of central purchase, where libraries could subscribe to journals which cost no more than 1.5 times as much as the average non-profit journals and allow their users free access.
What can one library do?
Marybeth Manning
Reset: a publisher's response to the changing economy
SPIE have taken the “unusual” step of reducing institutional subscription prices to their Digital Library by 10%, because they believed there would be sustained impact on scholarly publishing from changing publishing models and economic crises.
SPIE are a not for profit international society for optics and photonics: their Digital Library contains journals, conference proceedings and books. The subscription model is tiered (with different criteria for academic, government and corporate customers) and was originally based on print pricing.
Their objective is optimal dissemination of information, but it must be sustainable: a balance of reach and revenues. SPIE decided that their tier model was acceptable and understood, but penalised smaller institutions or large institutions with few relevant programmes. Corporations preferred price per use, and overall the cost per use for low tiers too high. Combined with environmental factors (the global economic crisis) they decided on overall price reductions on full DL subscriptions and topical segments (not including consortia or single subscriptions). They also added a 5th tier for the smallest organisations, and allowed introductory discounts to enable institutions to test interest. So far renewal levels have stayed strong, and they have good levels of new business, especially in lower tiers.
Does it apply to wider community?
SPIE wanted to shift their business model rather than have it shifted for them by factors outside their control, and believe that market growth and user demand will support reduced prices in future. They “want to be seen as part of the solution not the problem”.
But does this affect anything when big deals/mega publishers are so dominant? If libraries won't vote with their feet (or with their £) then who is to blame?
Carol Tenopir
University investments in the library: measuring the return
How does the library demonstrate its value to the university and its scholarship?
How can value be measured?
Derived measures: A way to show that the library contributes to the income generating activity of the university ie for every £ spent on the library, the university received £ in return. One way is to look at money spent on collections and then money coming back in grants.
Grant cycle => conduct research – write articles – write reports and proposals – obtain grants – conduct research etc. Libraries have already been connected to all steps aside from obtaining grants: this is key unknown.
University of Illinois did a phase 1 study funded by Elsevier which found $4.38 grant income for each $1 invested in the library (based on the % of faculty who rated citations in proposals as important in their success x % of proposals funded/library budget). Their findings have been published in a white paper.
Phase 2 expanded the methodology from phase 1 across 8 institutions in 8 countries – does it still apply? (But remember that ROI numbers do not tell the full story)
Big Deals and the Terrible Fix
Ted started by describing librarians' “shopping problems"
- Delegation: Librarians are making choices for consumers who aren't spending their own money – 'the university' pays
- Unreliable signals: Arguments for subscriptions are fervid but not always reliable
- Complexity: Libraries face all or nothing complex deals, and when there are large numbers of titles covering 100+ disciplines, who can say what it's worth?
- Monopoly: Journals represent price inelastic demand, i.e. demand doesn't vary according to price (much like the prescription drug industry)
The first big deals entailed a publisher (roughly speaking) working out the library's current spend, multiplying by a factor, and then offering e-access to all content in addition to the print for existing subscriptions.
How effective was this? Very, since the publisher knows that the library is willing to pay at least same as they did for their paper subscriptions, and giving them additional content costs the publisher nothing. Bundling works well from a seller's perspective: demand for bundles varies less than individual titles, and deters entry to the market by making it hard to add new competing journals. And of course, by the time the deal expires, the faculty are addicted to online access so the library has to negotiate a new deal.
Ted then gave the audience an economic prescription: namely, if you want to allocate resources efficiently, use a price system where users pay for what they get out of their own money, and therefore economise, ie a pay per view model. If informed users spend their own money on downloads, demand becomes more price elastic, monopolists are forced to cut prices, and authors prefer to publish in reasonably priced journals in order to get more readers.
Does it work for academics? Ask yourself: who else is better placed to decide value of article access?
Currently there's a big discrepancy between the average cost of a non-profit and for-profit article. The competition prediction suggests that if users pay, it would drive down prices to just over cost-price. There could be limited role of central purchase, where libraries could subscribe to journals which cost no more than 1.5 times as much as the average non-profit journals and allow their users free access.
What can one library do?
- Consider dropping big deal subscriptions to overpriced journals
- Maintain subscriptions to reasonably priced journals, at zero cost to end users
- Subsidise user-pays models (not to their full cost though)
Marybeth Manning
Reset: a publisher's response to the changing economy
SPIE have taken the “unusual” step of reducing institutional subscription prices to their Digital Library by 10%, because they believed there would be sustained impact on scholarly publishing from changing publishing models and economic crises.
SPIE are a not for profit international society for optics and photonics: their Digital Library contains journals, conference proceedings and books. The subscription model is tiered (with different criteria for academic, government and corporate customers) and was originally based on print pricing.
Their objective is optimal dissemination of information, but it must be sustainable: a balance of reach and revenues. SPIE decided that their tier model was acceptable and understood, but penalised smaller institutions or large institutions with few relevant programmes. Corporations preferred price per use, and overall the cost per use for low tiers too high. Combined with environmental factors (the global economic crisis) they decided on overall price reductions on full DL subscriptions and topical segments (not including consortia or single subscriptions). They also added a 5th tier for the smallest organisations, and allowed introductory discounts to enable institutions to test interest. So far renewal levels have stayed strong, and they have good levels of new business, especially in lower tiers.
Does it apply to wider community?
SPIE wanted to shift their business model rather than have it shifted for them by factors outside their control, and believe that market growth and user demand will support reduced prices in future. They “want to be seen as part of the solution not the problem”.
But does this affect anything when big deals/mega publishers are so dominant? If libraries won't vote with their feet (or with their £) then who is to blame?
Carol Tenopir
University investments in the library: measuring the return
How does the library demonstrate its value to the university and its scholarship?
How can value be measured?
- implicit value – downloads, usage
- explicit value – testimonials, researchers' purpose in using library services
- derived values – Return on Investment (ROI)
Derived measures: A way to show that the library contributes to the income generating activity of the university ie for every £ spent on the library, the university received £ in return. One way is to look at money spent on collections and then money coming back in grants.
Grant cycle => conduct research – write articles – write reports and proposals – obtain grants – conduct research etc. Libraries have already been connected to all steps aside from obtaining grants: this is key unknown.
University of Illinois did a phase 1 study funded by Elsevier which found $4.38 grant income for each $1 invested in the library (based on the % of faculty who rated citations in proposals as important in their success x % of proposals funded/library budget). Their findings have been published in a white paper.
Phase 2 expanded the methodology from phase 1 across 8 institutions in 8 countries – does it still apply? (But remember that ROI numbers do not tell the full story)
Library ROIs fell into three distinct tiers:
Universities' administration staff also said they wanted the library to help them:
Phase 3 (LibValue project) will be looking at more complex areas of library's value to teaching/learning, and social/professional activities. ARL will be involved in disseminating tools and measures. They recognise that they also need to look at new scholarly endeavours – e-science, collaborative scholarship, institutional repositories.
Conclusion
- STM research institutions: up to 15.5 to 1
- Research/teaching institutions STM/Humanities/Social Sciences: up to 3.4 to 1 (most common category)
- Research and teaching all disciplines, not so much emphasis on external funding: less than 1 to 1
Universities' administration staff also said they wanted the library to help them:
- attract/retain outstanding faculty (studies have shown a relationship between reading more library resources, publishing more, getting more grants: typical profile of 'star' faculty members)
- foster innovative research (bearing in mind that the number of articles cited will be much lower than those read)
- build research reputation of institution (some cases have shown positive relationship between library funding and amount of successful grants)
- promote seamless integration of the library with institutional research activities (some studies have shown a positive relationship between article downloads and research productivity)
Phase 3 (LibValue project) will be looking at more complex areas of library's value to teaching/learning, and social/professional activities. ARL will be involved in disseminating tools and measures. They recognise that they also need to look at new scholarly endeavours – e-science, collaborative scholarship, institutional repositories.
Conclusion
- It's possible to tie library e-collections to faculty productivity
- Libraries help generate grants income
- ROI for grants varies according to the mission and location of university
- Value can be measured in many ways