As academics obsess over how citations are used to measure performance, Rob Butler wonders whether they should borrow an idea from the music industry…
Geoscientist 20.8 August 2010
Metrics hysteria is creeping over UK universities as we gear up for the Research Excellence Framework. Publication records are the basis for appointments and promotion the world over. So there are new ways of designating which papers are important, moving from simple citations to adding Journal Impact Factors, and now valuing researchers by their so-called “H-index”. But what might we learn from other communities that obsess over individual performance and the impact of their products?
In September 2004, just as Journal Impact Factors were launched, Westlife’s
Flying Without Wings reached Number One. It may be a particularly forgettable boy-band ballad, but the song is important. It was the first to top the UK’s official singles chart on downloads. It wasn’t about “citations”. The music chart is not constructed from the number of times a song is mentioned in the music press or performed by a tribute band. As you listen on a Sunday afternoon to Radio 1’s chart show, you get a direct measure of a particular type of impact - sales. If people value the music they buy it. And mostly these days they do so by download.
Similarly, we increasingly download papers direct from publishers - either as a pdf, or by reading the full html version online. Download statistics track buyers. Surely these are a better measure of value than citations? I therefore thought it might be interesting to see how well citations track downloads for some of my own output. I took nine papers published in the Journal of the Geological Society between 1987 and 2008 and, with the help of the Publishing House, got the download statistics for a 30-month period. The comparison with citations is stark. According to Web of Science, the nine papers were cited 229 times, of which 37 accrued in the past 30 months. Their total number of downloads (html and pdf) for the same period was 4284. Even assuming authors read the papers they cite, the citations only represent about five percent of the demand for my research - or less than 1% in the 30 months. I suspect this imbalance is not unusual - and there is no correlation between my personal download chart and the citation record. So perhaps citations are poor measures of utility or value.
Now, I’m not arguing for weekly download hit-parades, but there may be advantages in the approach. Authors could choose to publish in journals that actively promote online access, just as successful artists can select better labels. This could improve online access for all, and would allow us to demonstrate that our work isn’t being read solely by a small cabal of the like-minded. If papers are used for teaching, industrial application, or to inform government, they will be downloaded - even though they may not be cited.
If you agree, vote now to get Simon Cowell onto the REF panel. I look forward to seeing the first “novelty” papers appearing for next Christmas. Or the dance remix of “Thrust Sequences”…