Thanks to one of my students — Addison Amiri — for pointing out this piece by @Richvn.
These journals aggressively curate their brands, in ways more conducive to selling subscriptions than to stimulating the most important research. Like fashion designers who create limited-edition handbags or suits, they know scarcity stokes demand, so they artificially restrict the number of papers they accept. The exclusive brands are then marketed with a gimmick called \”impact factor\” – a score for each journal, measuring the number of times its papers are cited by subsequent research. Better papers, the theory goes, are cited more often, so better journals boast higher scores. Yet it is a deeply flawed measure, pursuing which has become an end in itself – and is as damaging to science as the bonus culture is to banking.
Thanks to my colleague Diana Hicks for pointing this out to me.
The last line of the quotation strikes me as the most interesting point, one that deserves further development. The steering effect of metrics is well known (Weingart 2005). There’s growing resistance to the Journal Impact Factor. Although the persuasive comparison between researchers and bankers is itself over the top, the last line suggests — at least to me — a better way to critique the reliance on the Journal Impact Factor, as well as other attempts to measure research. It’s a sort of reverse Kant with an Illichian flavor, which I will formulate as a principle here, provided that everyone promises to keep in mind my attitude toward principles.
Here is one formulation of the principle: Measure researchers only in ways that recognize them as autonomous agents, never merely as means to other ends.
Here is another: Never treat measures as ends in themselves.
Once measures, which are instruments to the core, take on a life of their own, we have crossed the line that Illich calls the second watershed. That the Journal Impact Factor has in fact crossed that line is the claim made in the quote, above, though not using Illich’s language. The question we should be asking is how researchers can manage measures, rather than how we can measure researchers in order to manage them.
Peter Weingart. Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics Vol. 62, No. 1 (2005) 117-131.
To what degree is quantity being substituted for quality in today’s research assessment exercises? This strikes me as a symptom of the overvaluation of efficiency.
Higgs said he became \”an embarrassment to the department when they did research assessment exercises\”. A message would go around the department saying: \”Please give a list of your recent publications.\” Higgs said: \”I would send back a statement: \’None.\’ \”
Thanks to Lance Weihmuller for pointing me to the article.
This is definitely worth a look, whether you’re into the idea of post-publication peer review or not.
Do what you can today; help disrupt and redesign the scientific norms around how we assess, search, and filter science.
You know, I’m generally in favor of this idea — at least of the idea that we ought to redesign our assessment of research (science in the broad sense). But, as one might expect when speaking of design, the devil is in the details. It would be disastrous, for instance, to throw the baby of peer review out with the bathwater of bias.
I touch on the issue of bias in peer review in this article (coauthored with Steven Hrotic). I suggest that attacks on peer review are attacks on one of the biggest safeguards of academic autonomy here (coauthored with Robert Frodeman). On the relation between peer review and the values of autonomy and accountability, see: J. Britt Holbrook (2010). “Peer Review,” in The Oxford Handbook of Interdisciplinarity, Robert Frodeman, Julie Thompson Klein, Carl Mitcham, eds. Oxford: Oxford University Press: 321-32 and J. Britt Holbrook (2012). “Re-assessing the science – society relation: The case of the US National Science Foundation’s broader impacts merit review criterion (1997 – 2011),” in Peer Review, Research Integrity, and the Governance of Science – Practice, Theory, and Current Discussions. Robert Frodeman, J. Britt Holbrook, Carl Mitcham, and Hong Xiaonan. Beijing: People’s Publishing House: 328 – 62.
Let’s imagine that the top tier of higher education is actually not in the business of selling education. Instead, they are in what I would term the “talent identification” business. The real payoff for universities comes not from selling courses but rather from finding and nurturing talent and then waiting for payback in the form of contributions to their endowments.
Thanks to @pahndeepah for pointing this out to me. It’s definitely worth reading. I doubt too many of the business-types we now have running out universities will be able to stretch their imaginations this far.
Even if they can, however, they’ll be up against the habit of thought that sees students as consumers. The sort of loyalty needed to inspire former students to give back to their universities doesn’t mesh well with the idea that students are customers. At most, loyal customers may return to buy something else. But once you ‘buy’ the university degree, there’s not much incentive to go back and buy another. So why should students — seen as consumers — give back to their universities? They’ve already paid for their degree, after all.
This is an interesting read on the Open Access movement. Here’s the conclusion, with a quote from Michael Eisen that provides some food for thought.
In the end, his disdain isn’t directed at the publishers who hoard scientific knowledge so much as at his colleagues who let them get away with it. “One of the reasons advances in publishing don’t happen is that people are willing to live with all sorts of crap from journals in order to get the imprimatur the journal title has as a measure of the impact of their work,” Eisen says. “It’s easy to blame Elsevier, right? To think that there’s some big corporation that’s preventing scientists from doing the right thing. It’s just bullshit. Elsevier doesn’t prevent anyone from doing anything. Scientists do this themselves!”