Category Archives: Philosophy and Technology

Postmodern Research Evaluation? | 1 of ?

Posted on by

This will be the first is a series of posts tagged ‘postmodern research evaluation’ — a series meant to be critical and normative, expressing my own, subjective, opinions on the question.

Before I launch into any definitions, take a look at this on ‘Snowball Metrics‘. Reading only the first few pages should help orient you to where I am coming from. It’s a place from where I hope to prevent such an approach to metrics from snowballing — a good place, I think, for a snowball fight.

Read the opening pages of the snowball report. If you cannot see this as totalizing — in a very bad way — then we see things very differently. Still, I hope you read on, my friend. Perhaps I still have a chance to prevent the avalanche.

Nigel Warburton’s negative vision of what philosophy isn’t

Posted on by

Philosopher Nigel Warburton, of philosophy bites fame, has just resigned his academic post at the Open University to pursue other opportunities. The Philosopher’s Magazine conducts an extended interview with Warburton here. Much of what he reveals in this interview is both entertaining and, in my opinion, true.

But one aspect of the interview especially caught my attention. After offering several criticisms of academic philosophy today with which I’m in total agreement (in particular the tendency of hiring committees to hire clones of themselves rather than enhancing the diversity of the department), Warburton offers what he seems to view as the ultimate take down of academic philosophy. I quote this section in full, below. If you’ve been paying any attention to this blog or our posts at CSID, you’ll understand why, immediately.

He reserves particular venom for the REF, the Research Excellence Framework, a system of expert review which assesses research undertaken in UK higher education, which is then used to allocate future rounds of funding. A lot of it turns on the importance of research having a social, economic or cultural impact. It’s not exactly the sort of thing that philosophical reflection on, say, the nature of being qua being is likely to have. He leans into my recorder to make sure I get every word:

“One of the most disturbing things about academic philosophy today is the way that so many supposed gadflies and rebels in philosophy have just rolled over in the face of the REF – particularly by going along with the idea of measuring and quantifying impact,” he says, making inverted commas with his fingers, “a technical notion which was constructed for completely different disciplines. I’m not even sure what research means in philosophy. Philosophers are struggling to find ways of describing what they do as having impact as defined by people who don’t seem to appreciate what sort of things they do. This is absurd. Why are you wasting your time? Why aren’t you standing up and saying philosophy’s not like that? To think that funding in higher education in philosophy is going to be determined partly by people’s creative writing about how they have impact with their work. Just by entering into this you’ve compromised yourself as a philosopher. It’s not the kind of thing that Socrates did or that Hume did or that John Locke did. Locke may have had patrons, but he seemed to write what he thought rather than kowtowing to forces which are pushing on to us a certain vision, a certain view of what philosophical activities should be. Why are you doing this? I’m getting out. For those of you left in, how can you call yourselves philosophers? This isn’t what philosophy’s about.”

Please tell us how you really feel, Dr. Warburton.

In the US, we are not subject to the REF. But we are subject to many, many managerial requirements, including, if we seek grant funding, the requirement that we account for the impact of our research. We are, of course, ‘free’ to opt out of this sort of requirement simply by not seeking grant funding. Universities in the UK, however, are not ‘free’ to opt out of the REF. So, are the only choices open to ‘real’ philosophers worthy of the name resistance or removing oneself from the university, as Warburton has chosen?

I think not. My colleagues and I recently published an article in which we present a positive vision of academic philosophy today. A key aspect of our position is that the question of impact is itself a philosophical, not merely a technical, problem. Philosophers, in particular, should own impact rather than allowing impact to be imposed on us by outside authorities. The question of impact is a case study in whether the sort of account of freedom as non-domination offered by Pettit can be instantiated in a policy context, in addition to posited in political philosophy.

Being able to see impact as a philosophical question rests on being able to question the idea that the only sort of freedom worth having is freedom from interference. If philosophy matters to more than isolated individuals — even if connected by social media — then we have to realize that any philosophically rich conception of liberty must also include responsibility to others. Our notion of autonomy need not be reduced to the sort of non-interference that can only be guaranteed by separation (of the university from society, as Humboldt advocated, or of the philosopher from the university, as Warburton now suggests). Autonomy must be linked to accountability — and we philosophers should be able to tackle this problem without being called out as non-philosophers by someone who has chosen to opt out of this struggle.

Ross Mounce lays out easy steps towards open scholarship | Impact of Social Sciences

Posted on by

Excellent post with lots of good information here;

Easy steps towards open scholarship | Impact of Social Sciences.

There are some especially good thoughts about preprints.

Ross is right, I think, that using preprints is uncommon in the Humanities. For anyone interested in exploring the idea, I recommend the Social Epistemology Review and Reply Collective. Aside from being one of the few places to publish preprints in the Humanities, the SERRC preprints section also allows for extended responses to posted preprints, such as this one. The one major drawback (as Ross points out about sites such as Academia.edu) is that the SERRC doesn’t really archive preprints in the way that, say, a library would. Of course, if you happen to have an institutional repository, you can use that, as well.

Another site worth mentioning in this context is peerevaluation.org. I posted the same preprint on my page there. There are two interesting features of the peerevaluation.org site. One is that it uses interesting metrics, such as the ‘trust’ function. Similar to Facebook ‘likes’, but much richer, the ‘trust’ function allows users to build a visible reputation as a ‘trusted’ reviewer. What’s that, you ask? As a reviewer? Yes, and this is the second interesting feature of peerevaluation.org. It allows one to request reviews of posted papers. It also keeps track of who reviewed what. In theory, this could allow for something like ‘bottom-up’ peer review by genuine peers. One drawback of peerevaluation.org is that not enough people actually participate as reviewers. I encourage you to visit the site and serve as a reviewer to explore the possibilities.

As a humanist who would like to take advantage of preprints, both to improve my own work and for the citation advantage Ross notes, it’s difficult not to envy the situation in Physics and related areas (with arxiv). But how does such a tradition start? There are places one can use to publish preprints in the humanities. We need to start using them.

Quick thoughts on Challenges of Measuring Social Impact Using Altmetrics

Posted on by

As altmetric data can detect non-scholarly, non-traditional modes of research consumption, it seems likely that parties interested in social impact assessment via social reach may well start to develop altmetric-based analyses, to complement the existing approaches of case histories, and bibliometric analysis of citations within patent claims and published guidelines.

This and other claims worth discussing appear in this hot-off-the-presses (do we need another metaphor now?) article from Mike Taylor (@herrison):

The Challenges of Measuring Social Impact Using Altmetrics – Research Trends.

In response to the quote above, my own proposal would be to incorporate altmetrics into an overall narrative of impact. In other words, rather than have something like a ‘separate’ altmetric report, I’d rather have a way of appealing to altmetrics as one form of empirical evidence to back up claims of impact.

Although it is tempting to equate social reach (i.e., getting research into the hands of the public), it is not the same as measuring social impact. At the moment, altmetrics provides us with a way of detecting when research is being passed on down the information chains – to be specific, altmetrics detects sharing, or propagation events. However, even though altmetrics offers us a much wider view of how scholarly research is being accessed and discussed than bibliometrics, at the moment the discipline lacks an approach towards understanding the wider context necessary to understand both the social reach and impact of scholarly work.

Good point about the difference between ‘social reach’ and ‘social impact’. My suggestion for developing an approach to understanding the link between social reach and social impact would be something like this: social reach provides evidence of a sort of interaction. What’s needed to demonstrate social impact, however, is evidence of behavior change. Even if one cannot establish a direct causal relation between sharing and behavior change, demonstrating that one’s research ‘reached’ someone who then changed her behavior in ways consistent with what one’s paper says would generate a plausible narrative of impact.

 

Although altmetrics has the potential to be a valuable element in calculating social reach – with the hope this would provide insights into understanding social impact – there are a number of essential steps that are necessary to place this work on the same standing as bibliometrics and other forms of assessment.

My response to this may be predictable, but here goes anyway. I am all for improving the technology. Using Natural Language Processing, as Taylor suggests a bit later, sounds promising. But I think there’s a fundamental problem with comparing altmetrics to bibliometrics and trying to bring the former up to the standards of rigor of the latter. The problem is that this view privileges technology and technical rigor over judgment. Look, let’s make altmetrics as rigorous as we can. But please, let’s not make the mistake of thinking we’ve got the question of impact resolved once altmetrics have achieved the same sort of methodological rigor as bibliometrics! The question of impact can be answered better with help from technology. But to assume that technology can answer the question on its own (as if it existed independently of human beings, or we from it), is to fall into the trap of the technological fix.

San Francisco Declaration on Research Assessment — Well done, DORA

Posted on by

Anyone interested in research assessment should read this with care.

DORA.

It’s been presented in the media as an insurrection against the use of the Journal Impact Factor — and the Declaration certainly does … ehr … declare that the JIF shouldn’t be used to assess individual researchers or individual research articles. But this soundbite shouldn’t be used to characterize the totality of DORA, which is much broader than that.

Honestly, it took me a few days to go read it. After all, it’s uncontroversial in my mind that the JIF shouldn’t be used in this way. So, an insurrection against it didn’t strike me as all that interesting. I’m all for the use of altmetrics and — obviously, given our recent Nature correspondence (free to read here) — other inventive ways to tell the story of our impact.

But, and I cannot stress this enough, everyone should give DORA a careful read. I’m against jumping uncritically on the bandwagon in favor of Openness in all its forms. But I could find little reason not to sign, and myriad reasons to do so.

Well done, DORA.

The Document: an Open Letter From San Jose State U.’s Philosophy Department – Technology – The Chronicle of Higher Education

Posted on by

The Document: an Open Letter From San Jose State U.’s Philosophy Department – Technology – The Chronicle of Higher Education.

Remarkably well argued for a group of philosophers. And they are correct, in my opinion, about the crux of the matter.

Everyone interested in the future of the university should read the letter.

A Net Skeptic’s Conservative Manifesto – Reason.com

Posted on by

Steve Fuller sent me this review of Evgeny Morozov’s latest book:

A Net Skeptic’s Conservative Manifesto – Reason.com.

Of note was the comparison of Morozov with Oakeshott:

It remains unclear just how far Morozov would go to defeat “the cult of efficiency” that he says haunts us. Would he join Oakeshott in insisting that “the onus of proof, to show that the proposed change may be expected to be on the whole beneficial, rests with the would-be innovator”—in other words, applying the precautionary principle to technological change?  Morozov’s solutionism of “erratic appliances” and “technological troublemakers” would certainly constitute a preemptive, precautionary approach to digital regulation, should anyone attempt to apply them.

I haven’t read the book, yet. But this review makes me want to do so. Even if I don’t agree with Morozov’s conclusions — and I’m not saying I don’t or won’t — he seems to be asking some of the right questions.