PhyloPic Phryday Photo

Here are some other infrequently asked questions about impact that didn’t make it into the final cut of my piece at the LSE Impact of Social Sciences Blog.

Why conflate impact with benefit?

Put differently, why assume that all impacts are positive or benefits to society? Obviously, no one wants publicly supported research not to benefit the public. It’s even less palatable to consider that some publicly supported research may actually harm the public. But it’s wishful thinking to assume that all impacts are beneficial. Some impacts that initially appear beneficial may have negative consequences. And seemingly negative indicators might actually show that one is having an impact – even a positive one. I discuss this point with reference to Jeffrey Beall, recently threatened with a $1 billion lawsuit, here.

The question of impact is an opportunity to discuss such issues, rather than retreating into the shelter of imagined value-neutrality or objectivity. It was to spark this discussion that we generated a CSID-specific list – it is purposely idiosyncratic.

How can we maximize our impact?

I grant that ‘How can we maximize our impact?’ is a logistical question; but it incorporates a healthy dose of logos. Asking how to maximize our impacts should appeal to academics. We may be choosey about the sort of impact we desire and on whom; but no one wants to have minimal impact. We all desire to have as much impact as possible. Or, if we don’t, please get another job and let some of us who do want to make a difference have yours.

Wherefore impact?

For what reason are we concerned with the impact of scholarly communication? It’s the most fundamental question we should be asking and answering. We need to be mindful that whatever metrics we devise will have a steering effect on the course of scholarly communications. If we are going to steer scholarly communications, then we should discuss where we plan to go – and where others might steer us.

Developing indicators of the impact of scholarly communication is a massive technical challenge – but it’s also much simpler than that | Impact of Social Sciences

Posted on June 12, 2013 by jbrittholbrook

Developing indicators of the impact of scholarly communication is a massive technical challenge – but it’s also much simpler than that | Impact of Social Sciences.

In which I expand on ideas presented here and here.

Postmodern Research Evaluation? | 3 of ?

Posted on June 11, 2013 by jbrittholbrook

Snowball Metrics present as a totalizing grand narrative. For now, let me simply list some of the ways in which this is so, with little or only brief explanations.

Snowball metrics are a tool for commensuration, “designed to facilitate crossinstitutional benchmarking globally by ensuring that research management information can be compared with confidence” (p. 5 — with all references to page numbers in this PDF).
Snowball metrics are based on consensus: “Consensus on the ‘recipes’ for this first set of Snowball Metrics has been reached by a group of UK higher education institutions” (p. 8).
Despite the limited scope of the above consensus, however, Snowball Metrics are intended to be universal in scope, both in the UK “We expect that they will apply equally well to all UK institutions” and “to further support national and global benchmarking” (p. 8).
Snowball Metrics are presented as a recipe, one to be followed, of course. The word occurs 45 times in the 70 page PDF.
Other key words also appear numerous times: agree (including variations, such as ‘agreed’) appears 31 times; method (including variations, such as ‘methods’ or ‘methodology’) appears 22 times; manage (including variations) appears 15 times; impact appears 16 times, 11 times in terms of “Field-Weighted Citation Impact.”
Snowball Metrics are fair and “have tested methodologies that are freely available and can be generated by any organisation” (p. 7).
Snowball Metrics are ‘ours‘ — they are “defined and agreed by higher education institutions themselves, not imposed by organisations with potentially distinct aims” (p. 7).

To sum up, using their own words:

The approach is to agree a means to measure activities across the entire spectrum of research, at multiple levels of granularity: the Snowball Metrics Framework. (p. 7)

Coming in the next post (4 of ?), I present an alternative ‘framework’ — let’s call it Snowflake Indicators for now.

Postmodern Research Evaluation? | 2 of ?

Posted on June 11, 2013 by jbrittholbrook

First, let me say where I am coming from and what I mean by ‘postmodern’. I’m working from Lyotard’s simple “definition” of the term: “incredulity toward metanarratives” (from the introduction to The Postmodern Condition). One interesting question that arises from this definition is the scope of this incredulity — what counts, in other words, as a metanarrative?

Lyotard also distinguishes between what he calls ‘grand’ narratives and ‘little stories’ (les petits récits). Importantly, either a grand narrative or a little story can make the ‘meta’ move, which basically consists in telling a story about stories (where ‘story’ is understood broadly). Put differently, it is not the ‘meta’ toward which the postmodern reacts with incredulity. It is, rather, the totalizing character of the grand narrative that evinces doubt. By its very nature, the claim to have achieved certainty, to have told the whole story, undermines itself — at least from the postmodern perspective.

Of course, the grand narrative is always at pains to seek legitmation from outside itself, to demand recognition, to assert its own justice. Often, this takes the form of appeal to consensus — especially to a consensus of experts and authorities. The irony of the little stories is that they legitimate themselves precisely in not seeking hegemony over all the other stories. Not seeking jurisdiction over the whole, the little stories have the status — a venerable one — of ‘fables’. The little stories are told. We are told to accept the grand narrative.

Post 1 of ?

Postmodern Research Evaluation? | 1 of ?

Posted on June 11, 2013 by jbrittholbrook

This will be the first is a series of posts tagged ‘postmodern research evaluation’ — a series meant to be critical and normative, expressing my own, subjective, opinions on the question.

Before I launch into any definitions, take a look at this on ‘Snowball Metrics‘. Reading only the first few pages should help orient you to where I am coming from. It’s a place from where I hope to prevent such an approach to metrics from snowballing — a good place, I think, for a snowball fight.

Read the opening pages of the snowball report. If you cannot see this as totalizing — in a very bad way — then we see things very differently. Still, I hope you read on, my friend. Perhaps I still have a chance to prevent the avalanche.