Broader Impacts and Intellectual Merit: Paradigm Shift? | NOT UNTIL YOU CITE US!

On the one hand, this post on the VCU website is very cool.  It contains some interesting observations and what I think is some good advice for researchers submitting and reviewing NSF proposals.

Broader Impacts and Intellectual Merit: Paradigm Shift? | CHS Sponsored Programs.

On the other hand, this post also illustrates how researchers’ broader impacts go unnoticed.

One of my main areas of research is peer review at S&T funding agencies, such as NSF. I especially focus on the incorporation of societal impact criteria, such as NSF’s Broader Impacts Merit Review Criterion. In fact, I published the first scholarly article on broader impacts in 2005. My colleagues at CSID and I have published more than anyone else on this topic. Most of our research was sponsored by NSF.

I don’t just perform research on broader impacts, though. I take the idea that scholarly research should have some impact on the world seriously, and I try to put it into practice. One of the things I try to do is reach out to scientists, engineers, and research development professionals in an effort to help them improve the attention to broader impacts in the proposals they are working to submit to NSF. This past May, for instance, I traveled down to Austin to give a presentation at the National Organization for Research Development Professionals Conference (NORDP 2013). You can see a PDF version of my presentation at figshare.

If you look at the slides, you may recognize a point I made in a previous post, today. That point is that ‘intellectual merit’ and ‘broader impact’ are simply different perspectives on research. I made this point at NORDP 2013, as well, as you can see from my slides. Notice how they put the point on the VCU site:

Broader Impacts are just another aspect of their research that needs to be communicated (as opposed to an additional thing that must be “tacked on”).

I couldn’t have said it better myself. Or perhaps I could. Or perhaps I did. At NORDP 2013.

Again, VCU says:

Presenters at both conferences [they refer to something called NCURA, with that hyperlink, and to NORDP, with no hyperlink] have encouraged faculty to take the new and improved criteria seriously, citing that Broader Impacts are designed to answer accountability demands.  If Broader Impacts are not carefully communicated so that they are clear to all (even non-scientific types!), a door could be opened for more prescriptive national research priorities in the future—a move that would limit what types of projects can receive federal funding, and would ultimately inhibit basic research.

Unless someone else is starting to sound a lot like us, THIS IS OUR MESSAGE!

My point is not to claim ownership over these ideas. If I were worried about intellectual property, I could trademark a broader impacts catch phrase or something. My point is that if researchers don’t get any credit for the broader impacts of their research, they’ll be disinclined to engage in activities that might have broader impacts. I’m happy to share these ideas. How else could I expect to have a broader impact? I’ll continue to share them, even without attribution. That’s part of the code.

To clarify: I’m not mad. In fact, I’m happy to see these ideas on the VCU site (or elsewhere …). But would it kill them to add a hyperlink or two? Or a name? Or something? I’d be really impressed if they added a link to this post.

I’m also claiming this as evidence of the broader impacts of my research. I don’t have to contact any lawyers for that, do I?

UPDATE: BRIGITTE PFISTER, AUTHOR OF THE POST TO WHICH I DIRECTED MY DIATRIBE, ABOVE, HAS RESPONDED HERE. I APPRECIATE THAT A LOT. I ALSO LEFT A COMMENT APOLOGIZING FOR MY TONE IN THE ABOVE POST. IT’S AWAITING MODERATION; BUT I HOPE IT’S ACCEPTED AS IT’S MEANT — AS AN APOLOGY AND AS A SIGN OF RESPECT.

Calling All Thinkers — a plea for fostering diversity in thought

I am concerned that our educational system is blocking photorealistic visual thinkers like me from careers in science. Instead, we should encourage diversity in modes of thinking so that we aren’t losing the special talents of people who might contribute greatly to research and development by offering unique perspectives.

Calling All Thinkers | The Scientist Magazine®.

This is a good read. I do wish it included an image, though, beyond a photo of the author’s book cover. I often use images to try to make a similar point — that the question of impact is really a question of looking at different aspects of research, for instance. Here’s my go-to image for that claim:

Jastrow's Duck-Rabbit

It’s a simple point, but figuring out how to make it is difficult. I’m actually a big fan of the idea of involving the body somehow. I think this sort of perceptual shift is connected with our kinesthetic sense — it’s something we have to experience or feel.

If you care to get a sense of how I think, it’s shown by the ‘fact’ that I think these observations, above, are connected to both the Humboldtian idea of linking research and teaching and to my push to extend our thinking about altmetrics well beyond article-level metrics of even many different types of scholarly communication.

Altmetrics for “What Is Interdisciplinary Communication?”

Here is a link to the Altmetric Report for my recently published article “What Is Interdisciplinary Communication? Reflections on the Very Idea of Disciplinary Integration,” Synthese 190 (11): 1865-1879. DOI:10.1007/s11229-012-0179-7. There is also a preprint of the article available here.

Highlights of the Altmetric Report:

Compared to all articles in Synthese

So far Altmetric has tracked 78 articles from this journal. They typically receive a little less attention than average, with a mean score of 2.7 vs the global average of 3.6. This article has done particularly well, scoring higher than 99% of its peers. It’s actually the highest scoring article in this journal that we’ve seen so far.

In the
99%ile
Ranks
1st
All articles of a similar age

Older articles will score higher simply because they’ve had more time to accumulate mentions. To account for age we can compare this score to the 63,346 tracked articles that were published within six weeks on either side of this one in any journal. This article has done particularly well, scoring higher than 94% of its contemporaries.

In the
94%ile
Other articles of a similar age in Synthese

We’re also able to compare this article to 7 articles from the same journal and published within six weeks on either side of this one. This article has scored higher than all of them.

In the
99%ile
Ranks
1st
All articles

More generally, Altmetric has tracked 1,275,993 articles across all journals so far. Compared to these this article has done particularly well and is in the 96th percentile: it’s in the top 5% of all articles ever tracked by Altmetric.

In the
96%ile

Percentiles and ranks can obviously change with new publications. I also wonder whether one’s Altmetric score is not actually more a measure of one’s social media influence than it is a measure of the buzz surrounding an article — or maybe the two reduce to the same thing. But I sure like the sound of a number 1 ranking!

A pragmatic guide to monitoring and evaluating research communications using digital tools

Great blog for anyone interested in telling their own impact story!

Open Access and Its Enemies

I was thrilled to be invited to participate as a speaker in the University of North Texas Open Access Symposium 2013. It’s ongoing, and it’s being recorded; video of the presentations will be available soon. In the meantime, I’ve posted slides from my presentation on figshare.

I thought I’d add some thoughts here expounding on my presentation a bit and relating it to presentations given by my fellow panelists. I’m a proponent of open access, for several reasons. I think closed access, that is, encountering a pay wall when one goes to download a piece of research on is interested in reading, is unjust as well as inconvenient. The case for this claim can best be made with reference to two main points revolving around the question of intellectual property rights. Generally, in the case of closed access publications, authors are asked to sign away many, if not all, of their copy rights. Now, authors are free to negotiate terms with publishers, and we are free not to sign away our copy rights — but often the only choice with which we are left by many publishers is simply to take our work and publish it somewhere else.

Many otherwise ‘closed’ publishers will allow authors to retain all their copy rights for a fee (which varies from publisher to publisher) — this is known as the ‘author pays’ model of Gold OA (the latter term refers to OA publications in journals, as opposed to publications made OA via some sort of repository, which is known as Green OA). There is probably no better source for learning the terminology surrounding OA than Peter Suber’s website.

There is also the argument that when publicly funded research is published, the public should at least have free (gratis) access to the publication. Some publishers have argued against this on the grounds that they add value to the research by running the peer review process and formatting and archiving the article. They do perform these services, which do cost money (though peer review itself is done for free by academics). So, they argue that simply giving away their labor is unjust. If it is unjust to have the public pay again and unjust to ask publishers to give away the results of their labors, then, many argue, the ‘author pays’ model of OA makes the most sense. This, of course, ignores the fact of the free labor of academics in conducting peer review. (The labor of actually writing articles is arguably covered as part of an author’s base salary.) But even if authors are already paid to write the articles, it doesn’t follow that it’s just to ask them to pay again to have the articles made freely available once they are published.

Publishers, including Sage, are experimenting with different versions of the ‘author pays’ model of Gold OA. Jim Gilden was another member of my panel. He discussed Sage’s foray into OA, some of their innovations (including the interesting idea of having article-level editors who run the peer review process for individual articles, rather than for the journal as a whole), and some of the difficulties they have encountered. Among those difficulties is some sort of prejudice among potential authors — and members of promotion and tenure committees — against OA journals. This surprised me a little, but perhaps it should not have. One of the themes of my own talk is that ‘we’ academics are included among the enemies of open access. Our prejudice against OA publications is one indicator of this fact.

The other member of our panel was Jeffrey Beall, best known for Beall’s List of Predatory Open Access Publishers. Jeffrey talked about his list, including how and why it got started. That story is pretty simple: he started getting spam emails from publishers that didn’t quite feel right; as a cataloger, he did what came naturally and started keeping track; thus, Beall’s List. Things got more complicated after that. Many publishers appearing on Beall’s list are none too happy about it. Some have even threatened to sue Jeffrey — one for the sum of $1 billion! There are other, less publicized, sources of friction Jeffrey has encountered. He’s not too popular with his own university’s external/community relations folks. And he’s subject to a negative portrayal by many advocates of open access, who don’t appreciate the negative attention Beall’s list draws to the open access movement.

Criticism of Beall from publishers on his list is to be expected. In fact, it was serendipitous that I wrapped up the panel and ended my presentation with the slide of CSID’s list of ‘56 indicators of impact‘ — a list that includes negative indicators, such as provoking lawsuits. Jeffrey serves as a very good example of the sort of thing we are getting at with our list. The most important fact is that he has a narrative to account for why getting sued for $1 billion actually indicates that he’s having an impact. Unless a publisher were worried that Beall’s list would hurt their business, why would they threaten to sue?

Jeffrey and Jim were both excellent panel-mates for another reason. All three of us are not exactly full-fledged members of the open access enthusiasts club. Beall can’t be included, since his list can be interpreted as portraying not only specific publishers, but also the whole OA movement, in a negative light. Gilden can’t be included, since, well, he works for a for-profit-publisher. Those folks tend to be seen as more or less evil by many of the members of the OA crowd. (It was interesting to me to see the folks at Mendeley trying to — and having to — defend themselves on Twitter after Mendeley was bought by Elsevier, the evilist of all evil publishers.) And I? Well, as I said at the beginning of this post, I am an advocate of open access. But I am not an uncritical advocate, and I argue that a greater critical spirit needs to be embraced by many OA enthusiasts.

This was, in essence, the point of my talk. The text parts are pretty clear, I think. So let me focus here mostly on the images, and especially on the ‘Images of Impact’ slides. First, I explained how I derived my title from Popper’s The Open Society and Its Enemies. This seemed fitting not only because of the play on words, but also because I have come to see much of the struggle surrounding open access in terms of different conceptions of liberty or freedom. Popper’s emphasis on individual liberty was something I wanted to expand on, and I also linked it with Isaiah Berlin’s account of positive and negative conceptions of liberty. I also think Popper has an ambiguous relation to Neoliberalism. Popper was an original member of the Mont Pèlerin thought collective that many credit with the development and dissemination of Neoliberalism.

That Popper’s relation to Neoliberalism is unclear is an important point — and it’s another reason I chose him to introduce my talk. Part of what I wanted to suggest was that much of the open access movement is susceptible to being subsumed under a neoliberal agenda. After all, both use similar vocabularies — references to openness, to crowds, and to efficiency abound in both movements.

I didn’t really dwell on this point for long, though, in deference to the Symposium’s keynote speaker’s views on ‘neoliberalism‘. At the same time, I did want at least to reference Neoliberalism as one thing members of the open access movement need to be more aware of. I’m worried there’s something like a dogmatic enthusiasm that’s creeping into the OA crowd. Many of the reactions from within the OA enthusiast club against Jeffrey Beall (or against Mendeley) seem to me to betray an uncritical (and I mean un-self-critical) attitude. Similarly, I think it would be better for OA enthusiasts to examine carefully and to think critically about OA mandates and policies being considered now. Most, I fear, only think in terms like ‘any movement in the direction of more open access is good’. I just don’t believe that. In fact, I think it’s dangerous to think that way.

Sorry — on to my images of impact. I love altmetrics. I think that’s where you find many of the brightest advocates of open access. I also think the development of altmetrics is one of the areas most fraught with peril. After all, given the penchant of neoliberals for measurement-for-management-for-efficiency that goes by the name of ‘accountability’, it’s not difficult to see how numbers in general, and altmetrics in particular, might be co-opted by someone who wanted to do away with peer review and the protection that provides to the scholarly community. Talk of open, transparent, accountable government sounds great. But come on, folks, let’s please think about what that means. That drones are part of that plan ought to give us all pause. Altmetrics are the drones of the OA movement.

This is by no means to say that altmetrics are bad. I love altmetrics. I have said publicly that I think every journal should employ some form of article level metrics. They’re amazing. But they are also ripe for abuse — by publishers, by governments, and by academic administrators, among others. I just want altmetrics developers to recognize that possibility and to give it careful thought.

The development of altmetrics is not simply a technical issue. Nor are technologies morally or politically neutral. I suggested that we consider altmetrics (and perhaps OA in general) as a sociotechnical imaginary. I think the concept fits well here, especially linked to the idea of OA as a movement that entails an idea of positive freedom. There is a vision of the good associated with OA. Technology is supposed to help us along the road to achieving that good. Government policies are being enacted that may help. But we need to think critically about all of this rather than rushing forward in a burst of enthusiasm.

The great danger of positive freedom is that it can lead to coercion and even totalitarianism. The question is whether we can place a governor on our enthusiasm and limit our pursuit if positive freedom in a way that still allows for autonomy. I refer to Philip Petitt‘s notion of non-domination as potentially useful in this context. I also suggest that narrative can play a governing role. I do think we need some sort of localized (not totalizing) metanarrative about the relationship between the university and society (this is what I referred to in my talk in terms of a ‘republic of knowledge’). But narrative must also serve the role of de-totalizing in another sense. Narratives should be tied to articles and accompany article level metrics. We need to put the ‘account’ back into accountability, rather than simply focusing on the idea of counting.

So, to sum it all up: OA is good, but not an unqualified good; altmetrics are great, but they need to be accompanied by narratives. The end.

Negative Results – Not Just For Journal Articles?

David Bruggeman offers another twist on turning negatives into positives here. I’d like to add to this that it’s part of an ethos of not being afraid to make mistakes, even of valuing them. Some might refer to this as an entrepreneurial attitude.

David Bruggeman's avatarPasco Phronesis

There is a strong positive bias in how scientific knowledge is generated, written about, and measured.  It is easier to find research proving a hypothesis than replication studies that fail to confirm earlier findings.  It is easier to access explanations of why certain technologies came to be than studies about why we don’t have flying cars, or some other breakthrough promised to us through the magnificence of science and technology.  It’s an enormous hole in our understanding of the world, facilitated by the mores of the scientific reward system.

The same is true for metrics.  While the number of ways one can assess the impact of a particular paper is changing, many of the ‘alt’ metrics emerging are still thinking primarily in positive terms.  At least that’s the proposition of J. Britt Holbrook and some of his colleagues at the University of North Texas.  In a letter to Nature

View original post 185 more words