Postmodern Research Evaluation? | 1 of ?

Posted on June 11, 2013 by jbrittholbrook

This will be the first is a series of posts tagged ‘postmodern research evaluation’ — a series meant to be critical and normative, expressing my own, subjective, opinions on the question.

Before I launch into any definitions, take a look at this on ‘Snowball Metrics‘. Reading only the first few pages should help orient you to where I am coming from. It’s a place from where I hope to prevent such an approach to metrics from snowballing — a good place, I think, for a snowball fight.

Read the opening pages of the snowball report. If you cannot see this as totalizing — in a very bad way — then we see things very differently. Still, I hope you read on, my friend. Perhaps I still have a chance to prevent the avalanche.

Phylopic Phryday Photo

Hallucigenia by Caleb M. Brown

Broader Impacts and Intellectual Merit: Paradigm Shift? | NOT UNTIL YOU CITE US!

Posted on June 7, 2013 by jbrittholbrook

On the one hand, this post on the VCU website is very cool. It contains some interesting observations and what I think is some good advice for researchers submitting and reviewing NSF proposals.

Broader Impacts and Intellectual Merit: Paradigm Shift? | CHS Sponsored Programs.

On the other hand, this post also illustrates how researchers’ broader impacts go unnoticed.

One of my main areas of research is peer review at S&T funding agencies, such as NSF. I especially focus on the incorporation of societal impact criteria, such as NSF’s Broader Impacts Merit Review Criterion. In fact, I published the first scholarly article on broader impacts in 2005. My colleagues at CSID and I have published more than anyone else on this topic. Most of our research was sponsored by NSF.

I don’t just perform research on broader impacts, though. I take the idea that scholarly research should have some impact on the world seriously, and I try to put it into practice. One of the things I try to do is reach out to scientists, engineers, and research development professionals in an effort to help them improve the attention to broader impacts in the proposals they are working to submit to NSF. This past May, for instance, I traveled down to Austin to give a presentation at the National Organization for Research Development Professionals Conference (NORDP 2013). You can see a PDF version of my presentation at figshare.

If you look at the slides, you may recognize a point I made in a previous post, today. That point is that ‘intellectual merit’ and ‘broader impact’ are simply different perspectives on research. I made this point at NORDP 2013, as well, as you can see from my slides. Notice how they put the point on the VCU site:

Broader Impacts are just another aspect of their research that needs to be communicated (as opposed to an additional thing that must be “tacked on”).

I couldn’t have said it better myself. Or perhaps I could. Or perhaps I did. At NORDP 2013.

Again, VCU says:

Presenters at both conferences [they refer to something called NCURA, with that hyperlink, and to NORDP, with no hyperlink] have encouraged faculty to take the new and improved criteria seriously, citing that Broader Impacts are designed to answer accountability demands. If Broader Impacts are not carefully communicated so that they are clear to all (even non-scientific types!), a door could be opened for more prescriptive national research priorities in the future—a move that would limit what types of projects can receive federal funding, and would ultimately inhibit basic research.

Unless someone else is starting to sound a lot like us, THIS IS OUR MESSAGE!

My point is not to claim ownership over these ideas. If I were worried about intellectual property, I could trademark a broader impacts catch phrase or something. My point is that if researchers don’t get any credit for the broader impacts of their research, they’ll be disinclined to engage in activities that might have broader impacts. I’m happy to share these ideas. How else could I expect to have a broader impact? I’ll continue to share them, even without attribution. That’s part of the code.

To clarify: I’m not mad. In fact, I’m happy to see these ideas on the VCU site (or elsewhere …). But would it kill them to add a hyperlink or two? Or a name? Or something? I’d be really impressed if they added a link to this post.

I’m also claiming this as evidence of the broader impacts of my research. I don’t have to contact any lawyers for that, do I?

UPDATE: BRIGITTE PFISTER, AUTHOR OF THE POST TO WHICH I DIRECTED MY DIATRIBE, ABOVE, HAS RESPONDED HERE. I APPRECIATE THAT A LOT. I ALSO LEFT A COMMENT APOLOGIZING FOR MY TONE IN THE ABOVE POST. IT’S AWAITING MODERATION; BUT I HOPE IT’S ACCEPTED AS IT’S MEANT — AS AN APOLOGY AND AS A SIGN OF RESPECT.

Calling All Thinkers — a plea for fostering diversity in thought

Posted on June 7, 2013 by jbrittholbrook

I am concerned that our educational system is blocking photorealistic visual thinkers like me from careers in science. Instead, we should encourage diversity in modes of thinking so that we aren’t losing the special talents of people who might contribute greatly to research and development by offering unique perspectives.

Calling All Thinkers | The Scientist Magazine®.

This is a good read. I do wish it included an image, though, beyond a photo of the author’s book cover. I often use images to try to make a similar point — that the question of impact is really a question of looking at different aspects of research, for instance. Here’s my go-to image for that claim:

It’s a simple point, but figuring out how to make it is difficult. I’m actually a big fan of the idea of involving the body somehow. I think this sort of perceptual shift is connected with our kinesthetic sense — it’s something we have to experience or feel.

If you care to get a sense of how I think, it’s shown by the ‘fact’ that I think these observations, above, are connected to both the Humboldtian idea of linking research and teaching and to my push to extend our thinking about altmetrics well beyond article-level metrics of even many different types of scholarly communication.

Altmetrics for “What Is Interdisciplinary Communication?”

Posted on June 6, 2013 by jbrittholbrook

Here is a link to the Altmetric Report for my recently published article “What Is Interdisciplinary Communication? Reflections on the Very Idea of Disciplinary Integration,” Synthese 190 (11): 1865-1879. DOI:10.1007/s11229-012-0179-7. There is also a preprint of the article available here.

Highlights of the Altmetric Report:

Compared to all articles in Synthese

So far Altmetric has tracked 78 articles from this journal. They typically receive a little less attention than average, with a mean score of 2.7 vs the global average of 3.6. This article has done particularly well, scoring higher than 99% of its peers. It’s actually the highest scoring article in this journal that we’ve seen so far.

In the
99%ile

Ranks
1st

All articles of a similar age

Older articles will score higher simply because they’ve had more time to accumulate mentions. To account for age we can compare this score to the 63,346 tracked articles that were published within six weeks on either side of this one in any journal. This article has done particularly well, scoring higher than 94% of its contemporaries.

In the
94%ile

Other articles of a similar age in Synthese

We’re also able to compare this article to 7 articles from the same journal and published within six weeks on either side of this one. This article has scored higher than all of them.

In the
99%ile

Ranks
1st

All articles

More generally, Altmetric has tracked 1,275,993 articles across all journals so far. Compared to these this article has done particularly well and is in the 96th percentile: it’s in the top 5% of all articles ever tracked by Altmetric.

In the
96%ile

Percentiles and ranks can obviously change with new publications. I also wonder whether one’s Altmetric score is not actually more a measure of one’s social media influence than it is a measure of the buzz surrounding an article — or maybe the two reduce to the same thing. But I sure like the sound of a number 1 ranking!