The tracking of the use of research has become central to the measurement of research impact. While historically this tracking has meant using citations to published papers, the results are old, biased, and inaccessible – and stakeholders need current data to make funding decisions. We can do much better. Today’s users of research interact with that research online. This leaves an unprecedented data trail that can provide detailed data on the attention that specific research outputs, institutions, or domains receive.
However, while the promise of real time information is tantalizing, the collection of this data is outstripping our knowledge of how best to use it, our understanding of its utility across differing research domains and our ability to address the privacy and confidentiality issues. This is particularly true in the field of Humanities and Social Sciences, which have historically been under represented in the collection of scientific corpora of citations, and which are now under represented by the tools and analysis approaches being developed to track the use and attention received by STM research outputs.
We will convene a meeting that combines a discussion of the state of the art in one way in which research impact can be measured – Article Level and Altmetrics – with a critical analysis of current gaps and identification of ways to address them in the context of Humanities and Social Sciences.
We would like to announce that Altmetric have begun tracking mentions of academic articles on Chinese microblogging site Sina Weibo, and the data will shortly be fully integrated into existing Altmetric tools.
The mentions collated will be visible to users via the Altmetric Explorer, a web-based application that allows users to browse the online mentions of any academic article, and, where appropriately licensed, via the article metrics data on publisher platforms.
Launched in 2009, Sina Weibo has become one of the largest social media sites in China, and is most often likened to Twitter. Integrating this data means that Altmetric users will now be able to see a much more global view of the attention an article has received. Altmetric is currently the only article level metrics provider to offer this data.
I’m looking forward to moderating a panel on day 1 of the AAAS Forum on Science and Technology Policy.
2:00 Current Issues in S&T Policy (Breakout Sessions)(A) Measuring the Impacts of Science
• What are the policy relevant challenges, tools, and approaches to measuring the social impact of scientific research? • How can improved indicators capture change in science, technology, and innovation? • Are altmetrics the solution to measuring social impacts?Moderator: J. Britt Holbrook, Visiting Assistant Professor, School of Public Policy, Georgia Institute of Technology; and Member, AAAS Committee on Scientific Freedom and ResponsibilityKaye Husbands Fealing, Professor, Center for Science, Technology and Environmental Policy, Humphrey School of Public Affairs, University of Minnesota; Senior Study Director, National Academy of Sciences, Committee on National Statistics; and Member, AAAS Committee on Science, Engineering, and Public PolicyGil Omenn, Director, Center for Computational Medicine and Bioinformatics, University of MichiganMike Taylor, Research Specialist, Elsevier Labs
These journals aggressively curate their brands, in ways more conducive to selling subscriptions than to stimulating the most important research. Like fashion designers who create limited-edition handbags or suits, they know scarcity stokes demand, so they artificially restrict the number of papers they accept. The exclusive brands are then marketed with a gimmick called \”impact factor\” – a score for each journal, measuring the number of times its papers are cited by subsequent research. Better papers, the theory goes, are cited more often, so better journals boast higher scores. Yet it is a deeply flawed measure, pursuing which has become an end in itself – and is as damaging to science as the bonus culture is to banking.
Thanks to my colleague Diana Hicks for pointing this out to me.
The last line of the quotation strikes me as the most interesting point, one that deserves further development. The steering effect of metrics is well known (Weingart 2005). There’s growing resistance to the Journal Impact Factor. Although the persuasive comparison between researchers and bankers is itself over the top, the last line suggests — at least to me — a better way to critique the reliance on the Journal Impact Factor, as well as other attempts to measure research. It’s a sort of reverse Kant with an Illichian flavor, which I will formulate as a principle here, provided that everyone promises to keep in mind my attitude toward principles.
Here is one formulation of the principle: Measure researchers only in ways that recognize them as autonomous agents, never merely as means to other ends.
Here is another: Never treat measures as ends in themselves.
Once measures, which are instruments to the core, take on a life of their own, we have crossed the line that Illich calls the second watershed. That the Journal Impact Factor has in fact crossed that line is the claim made in the quote, above, though not using Illich’s language. The question we should be asking is how researchers can manage measures, rather than how we can measure researchers in order to manage them.
Peter Weingart. Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics Vol. 62, No. 1 (2005) 117-131.
The contest was an opportunity for members of the scientific community to showcase the broader impacts of the biological sciences, including informing natural resources management, addressing climate change, and advancing foundational knowledge. The photos will be used to help the public and policymakers to better understand the value of biological research and education.
This is definitely worth a look, whether you’re into the idea of post-publication peer review or not.