Broader Impacts and Intellectual Merit: Paradigm Shift? | NOT UNTIL YOU CITE US!

On the one hand, this post on the VCU website is very cool.  It contains some interesting observations and what I think is some good advice for researchers submitting and reviewing NSF proposals.

Broader Impacts and Intellectual Merit: Paradigm Shift? | CHS Sponsored Programs.

On the other hand, this post also illustrates how researchers’ broader impacts go unnoticed.

One of my main areas of research is peer review at S&T funding agencies, such as NSF. I especially focus on the incorporation of societal impact criteria, such as NSF’s Broader Impacts Merit Review Criterion. In fact, I published the first scholarly article on broader impacts in 2005. My colleagues at CSID and I have published more than anyone else on this topic. Most of our research was sponsored by NSF.

I don’t just perform research on broader impacts, though. I take the idea that scholarly research should have some impact on the world seriously, and I try to put it into practice. One of the things I try to do is reach out to scientists, engineers, and research development professionals in an effort to help them improve the attention to broader impacts in the proposals they are working to submit to NSF. This past May, for instance, I traveled down to Austin to give a presentation at the National Organization for Research Development Professionals Conference (NORDP 2013). You can see a PDF version of my presentation at figshare.

If you look at the slides, you may recognize a point I made in a previous post, today. That point is that ‘intellectual merit’ and ‘broader impact’ are simply different perspectives on research. I made this point at NORDP 2013, as well, as you can see from my slides. Notice how they put the point on the VCU site:

Broader Impacts are just another aspect of their research that needs to be communicated (as opposed to an additional thing that must be “tacked on”).

I couldn’t have said it better myself. Or perhaps I could. Or perhaps I did. At NORDP 2013.

Again, VCU says:

Presenters at both conferences [they refer to something called NCURA, with that hyperlink, and to NORDP, with no hyperlink] have encouraged faculty to take the new and improved criteria seriously, citing that Broader Impacts are designed to answer accountability demands.  If Broader Impacts are not carefully communicated so that they are clear to all (even non-scientific types!), a door could be opened for more prescriptive national research priorities in the future—a move that would limit what types of projects can receive federal funding, and would ultimately inhibit basic research.

Unless someone else is starting to sound a lot like us, THIS IS OUR MESSAGE!

My point is not to claim ownership over these ideas. If I were worried about intellectual property, I could trademark a broader impacts catch phrase or something. My point is that if researchers don’t get any credit for the broader impacts of their research, they’ll be disinclined to engage in activities that might have broader impacts. I’m happy to share these ideas. How else could I expect to have a broader impact? I’ll continue to share them, even without attribution. That’s part of the code.

To clarify: I’m not mad. In fact, I’m happy to see these ideas on the VCU site (or elsewhere …). But would it kill them to add a hyperlink or two? Or a name? Or something? I’d be really impressed if they added a link to this post.

I’m also claiming this as evidence of the broader impacts of my research. I don’t have to contact any lawyers for that, do I?

UPDATE: BRIGITTE PFISTER, AUTHOR OF THE POST TO WHICH I DIRECTED MY DIATRIBE, ABOVE, HAS RESPONDED HERE. I APPRECIATE THAT A LOT. I ALSO LEFT A COMMENT APOLOGIZING FOR MY TONE IN THE ABOVE POST. IT’S AWAITING MODERATION; BUT I HOPE IT’S ACCEPTED AS IT’S MEANT — AS AN APOLOGY AND AS A SIGN OF RESPECT.

Nigel Warburton’s negative vision of what philosophy isn’t

Philosopher Nigel Warburton, of philosophy bites fame, has just resigned his academic post at the Open University to pursue other opportunities. The Philosopher’s Magazine conducts an extended interview with Warburton here. Much of what he reveals in this interview is both entertaining and, in my opinion, true.

But one aspect of the interview especially caught my attention. After offering several criticisms of academic philosophy today with which I’m in total agreement (in particular the tendency of hiring committees to hire clones of themselves rather than enhancing the diversity of the department), Warburton offers what he seems to view as the ultimate take down of academic philosophy. I quote this section in full, below. If you’ve been paying any attention to this blog or our posts at CSID, you’ll understand why, immediately.

He reserves particular venom for the REF, the Research Excellence Framework, a system of expert review which assesses research undertaken in UK higher education, which is then used to allocate future rounds of funding. A lot of it turns on the importance of research having a social, economic or cultural impact. It’s not exactly the sort of thing that philosophical reflection on, say, the nature of being qua being is likely to have. He leans into my recorder to make sure I get every word:

“One of the most disturbing things about academic philosophy today is the way that so many supposed gadflies and rebels in philosophy have just rolled over in the face of the REF – particularly by going along with the idea of measuring and quantifying impact,” he says, making inverted commas with his fingers, “a technical notion which was constructed for completely different disciplines. I’m not even sure what research means in philosophy. Philosophers are struggling to find ways of describing what they do as having impact as defined by people who don’t seem to appreciate what sort of things they do. This is absurd. Why are you wasting your time? Why aren’t you standing up and saying philosophy’s not like that? To think that funding in higher education in philosophy is going to be determined partly by people’s creative writing about how they have impact with their work. Just by entering into this you’ve compromised yourself as a philosopher. It’s not the kind of thing that Socrates did or that Hume did or that John Locke did. Locke may have had patrons, but he seemed to write what he thought rather than kowtowing to forces which are pushing on to us a certain vision, a certain view of what philosophical activities should be. Why are you doing this? I’m getting out. For those of you left in, how can you call yourselves philosophers? This isn’t what philosophy’s about.”

Please tell us how you really feel, Dr. Warburton.

In the US, we are not subject to the REF. But we are subject to many, many managerial requirements, including, if we seek grant funding, the requirement that we account for the impact of our research. We are, of course, ‘free’ to opt out of this sort of requirement simply by not seeking grant funding. Universities in the UK, however, are not ‘free’ to opt out of the REF. So, are the only choices open to ‘real’ philosophers worthy of the name resistance or removing oneself from the university, as Warburton has chosen?

I think not. My colleagues and I recently published an article in which we present a positive vision of academic philosophy today. A key aspect of our position is that the question of impact is itself a philosophical, not merely a technical, problem. Philosophers, in particular, should own impact rather than allowing impact to be imposed on us by outside authorities. The question of impact is a case study in whether the sort of account of freedom as non-domination offered by Pettit can be instantiated in a policy context, in addition to posited in political philosophy.

Being able to see impact as a philosophical question rests on being able to question the idea that the only sort of freedom worth having is freedom from interference. If philosophy matters to more than isolated individuals — even if connected by social media — then we have to realize that any philosophically rich conception of liberty must also include responsibility to others. Our notion of autonomy need not be reduced to the sort of non-interference that can only be guaranteed by separation (of the university from society, as Humboldt advocated, or of the philosopher from the university, as Warburton now suggests). Autonomy must be linked to accountability — and we philosophers should be able to tackle this problem without being called out as non-philosophers by someone who has chosen to opt out of this struggle.

‘Pure hype of pure research helps no one’ says Sarewitz; what this says about freedom

I tend to agree with a lot of what Dan Sarewitz argues here:

Pure hype of pure research helps no one : Nature News & Comment.

But I also want to suggest that there’s an argument to be made against the High Quality Research Act that goes beyond Sarewitz’s claim that it helps no one.

To be fair, that’s just the headline. Sarewitz also claims something I think is a bit more controversial — that the HQRA is really nothing to get too worried about. Not only does it help no one, but also it doesn’t hurt anyone.

This strikes me as mistaken. I’ll try to articulate why in terms of the distinction between negative and positive freedom I’ve been exploring. Here goes.

First, I agree that the HQRA helps no one.  But it’s not just that the HQRA is redundant — though this is certainly true. It’s also that it doesn’t allow us to do anything more to demonstrate our accountability, as I think the Broader Impacts Criterion does. In other words, it doesn’t increase anyone’s positive freedom.

Second, it actually decreases our negative freedom. By requiring NSF to re-certify what the merit review process already certifies (at least when it’s working as designed), this ‘added layer of accountability’ actually just increases the kind of bureaucratic red tape we should be trying to decrease if we’re interested in an efficient government. This makes about as much sense as the Florida Blue Ribbon Task Force’s suggestion to charge more for classes in majors that supposedly won’t result in better jobs for graduates. Majors that result in higher paying jobs actually should be in greater demand, and so should cost more, not less. But not according to the Blue Ribbon Task Force (see pp 22-3).

Finally, I think the HQRA might be a case study in how to reconcile notions of positive and negative freedom — or at least how to think of both ideas of liberty as possibly working together. It’s sort of a test. Sometimes, a policy that might increase our positive freedom can be seen as decreasing our negative freedom. I think the NSF’s Broader Impacts Criterion is a case in point. Yes, it places an additional burden on researchers, and so in this sense it is a limitation on their negative freedom. But it also increases their positive freedom, so that a trade-off is possible. The HQRA, on the other hand, decreases our negative freedom without also increasing our positive freedom. Any policy that doesn’t increase either our positive or our negative freedom is highly questionable — or so I am suggesting.

Autonomy, negative and positive freedom, and broader impacts

I’ve been continuing to think about the topic I broached in a previous post. There, I described what I termed ‘old’ and ‘new’ ways of thinking about autonomy and accountability. Of course, my ‘new’ ways are not all that new.

Here I want to sketch very briefly some recent thoughts I’ve had as a result of teaching my class this past semester at UNT in ‘Science, Technology, and Society’, my recent attendance at the ‘Broader Impacts Infrastructure Summit‘ at the University of Missouri, and recent discussions with colleagues. I want to thank my students for forcing me to try to get my thoughts in enough order to be able to stand up in front of them and teach. I also want to thank Steve Fuller for turning me on to some new reading material and Bob Frodeman and Adam Briggle, with whom I tend to discuss everything.

I don’t want to blame any of them for these thoughts, of course. But I mention them, in part, to indicate ways in which my own thinking is influenced by these ‘non-research’ activities. In particular, my teaching this past semester has hardly been separable from my research. This notion that teaching and research can be — and should be — mutually reinforcing in this way is one of two main Humboldtian theses about the university. For short, let’s call it the integration of research and teaching (though I’m on record about my suspicion of the term ‘integration’).

The other main Humboldtian thesis about the university is that it should be autonomous from the state. This same notion was expressed in the post WWII era by Vannevar Bush, whose “Science — The Endless Frontier” forms much of the conceptual basis of science policy. In my previous post on autonomy and accountability, I argued that we needed to get past an account of freedom as freedom from constraint by the state. What I did not say in the last post is that this is precisely the account of freedom that both Humboldt and Bush share.

This notion of freedom is very much like what Isaiah Berlin describes as ‘negative freedom’. Under such a negative account of freedom, any interference on the part of the state with the sort of research that’s performed at universities would be an assault on freedom. I think this is the sort of freedom most academics believe in, whether they know it or not. The idea is that academics should be free from outside interference. Only free in this sense will they be allowed to roam wherever their curiosity would take them.

According to my ‘new’ account of autonomy, however, freedom means self-determination rather than freedom from restraint. This notion actually corresponds in many ways to what Berlin describes in the essay cited above as ‘positive liberty’. Under this conception of freedom, requirements such as the NSF’s Broader Impacts Merit Review Criterion need not curtail academic freedom, as long as they don’t take away the academic’s ability to self-legislate. Instead, such requirements could be seen under the rubric of ‘interference without domination’ as outlined by Philip Pettit in his account of Republicanism (thanks to Steve Fuller for that reference — have been trying to read some about Pettit and Republican political theory in the past day or two).

I am not quite comfortable with transferring ideas straight from political philosophy to the realm of knowledge policy. But I’m not worried here about making a mistake in the details. I think there’s something helpful in seeing things this way.

One more point that may seem unrelated — though it isn’t, I just won’t be able to make all the connections here. At the Broader Impacts Summit, it became clear that there were two ‘camps’ forming. Again, I’m going to speak loosely. One camp wanted to claim broader impacts as its area of expertise — let’s call them the BI-Experts, though I believe most of them would consider themselves experts in science education and outreach. What they were pushing, albeit in still-developing form, was to make broader impacts activities a side show to the real research. It would be a rigorous sideshow, mind you. But broader impacts activities ought to be the realm of BI-Experts, rather than scientists, who can’t tell outreach from a hole in the ground.

The other group — and interestingly, it tended to include former and current NSF program officers — had what I labeled a Humboldtian view of broader impacts. The idea was that intellectual merit and broader impacts should be integrated in the project (and perhaps in the person of the Principal Investigator). Without wanting to limit broader impacts to teaching, this view is consistent with the Humboldtian position on the integration of research and teaching.

However, the Humboldtian take on broader impacts actually depends on a positive notion of freedom. If one were to take a negative view of freedom (as I think Humboldt does), then one would be more inclined toward the BI-Expert approach. Why? The BI-Experts will run interference for the scientists, who will then be free to perform their research without having to be concerned with broader impacts.

Obama calls for peer-review autonomy : Nature News Blog

Obama calls for peer-review autonomy : Nature News Blog.

Here’s Obama’s take on autonomy, accountability, and peer review.

U.S. Lawmaker Proposes New Criteria for Choosing NSF Grants – ScienceInsider

U.S. Lawmaker Proposes New Criteria for Choosing NSF Grants

via U.S. Lawmaker Proposes New Criteria for Choosing NSF Grants – ScienceInsider.

This is just the latest in a long saga. Most of the articles on NSF’s efforts to deal with such accountability demands by means of its Broader Impacts Merit Review Criterion are available here.