This short post from Andy Revkin combines several of my interests: fracking, peer review, and scholarly communication.
This paper describes the results of a survey regarding the incorporation of societal impacts considerations into the peer review of grant proposals submitted to public science funding bodies. The survey investigated perceptions regarding the use of scientific peers to judge not only the intrinsic scientific value of proposed research, but also its instrumental value to society. Members of the scientific community have expressed – some more stridently than others – resistance to the use of such societal impact considerations. We sought to understand why. Results of the survey suggest that such resistance may be due to a lack of desire rather than a lack of confidence where judging impacts is concerned. In other words, it may be less that scientists feel unable to judge broader societal impacts and more that they are unwilling to do so.
Absent from the many analyses and discussions of scientific peer review are two intangible but very important byproducts: 1) feedback to the applicant and 2) exposure of the reviewers to new hypotheses, techniques, and approaches. Both of these phenomena have a virtual mentoring effect that helps move science forward. Such learning can occur as a consequence of both manuscript review and grant application review, but the review of grant applications, by its very nature, is more iterative and impacts the direction in which research moves very early in the investigation.
There are at least two funding agencies that recognize this phenomenon in the actual design of their peer review processes, so they deserve mention. The idea is to include non-academics as peer reviewers precisely to effect the sort of co-production of knowledge the article above suggests.
The second is the US Congressionally Directed Medical Research Program. Details of their peer review process are available on their website here.
Scientists don’t like me. Or, at least, they don’t show any evidence of liking what I have to say about NSF’s Broader Impacts Merit Review Criterion. Last week, I blogged this ScienceInsider interview (here and on the CSID blog) with an unnamed congressional aide connected with Rep. Lamar Smith and his efforts to add “an extra layer of accountability” to NSF’s Merit Review Process.
I also left a couple of comments in the comments section under the article itself. It’s possible for readers of ScienceInsider to press buttons to indicate their agreement — or not — with comments. The site then tracks the number of likes or dislikes (expressed by pressing up or down carrots), displays them with each comment, and moves those comments with the most likes up to the top.
Guess whose comments are dead last in line?
Here are the two most-liked comments:
Studying dairy production in China is a very poor choice for an example of what constitutes a bad grant. It has direct relevance to something most people in America consume every day. It could reduce cost for millions, increase food safety, improve the quality or nutrient density of a commonly consumed item, etc. Every time I hear a story on Fox about a “wasteful” study, I can usually think of ten ways it could benefit people and industry here. Somehow I think the time would be better spent putting in an “additional layer” to cover pentagon spending.
A lot of smoke is being blown by Rep. Lamar Smith’s aide. The aide’s answers are slick and cover’s the real intent of the bill- to politicize the sciences through selective funding or defunding of areas of study Republicans do not like. The most egregious example would be the ban on studying gun violence in America.
Each of them has received twelve likes.
I suppose if I were simply to say that Congress is out to politicize science or that Smith is out of his depth or that scientists should be left alone to pursue research however they wish, scientists might like that. But I’m willing to give Smith the benefit of the doubt, at this point. My contention is that he (or his aide) doesn’t yet understand the revisions to NSF’s Merit Review Process. If he did, then I think he’d see that accountability is already built into the process. I think Smith should not introduce the High Quality Research Act, but instead should seek to monitor how scientists respond to the new Broader Impacts Criterion.
But there’s a real problem with what I’m suggesting. And it’s not that Smith is a Republican out to get science. The problem is that scientists themselves don’t understand the Broader Impacts Criterion. They don’t understand that this is their last, best hope to preserve their academic autonomy while meeting accountability demands. And they don’t want to hear it, either.
To see my comments on the ScienceInsider interview, simply follow this link and scroll to the bottom of the page.
ScienceInsider‘s take on Obama’s speech at the National Academy of Sciences:
Academics have tended to view autonomy as freedom from constraint by the state. We want to be able to go wherever our thoughts lead us, without ‘outside’ interference. This interpretation of autonomy as freedom from constraint is a negative definition, appealing to what is absent. It says, ‘Hands off!’
Politicians have tended to view accountability only in economic terms — that is, accountability has been reduced to accounting and conceived narrowly as something like return on investment (ROI).
Under such ‘old thinking’, autonomy is opposed to accountability, since accountability conceived as demonstrable ROI puts constraints on the autonomy of researchers conceived as freedom from such demands.
Autonomy means self-legislation, rather than freedom from constraint. Under this ‘new thinking’ on autonomy, the point is not for academics to be free from all constraint (the negative definition rooted in old thinking), but rather for academics to give themselves whatever constraints they are subject to.
Accountability means being able to give an account, in the Socratic sense of the term. This is by no means limited to a notion of ROI, though such may be included in the account one is expected to give.
Under this ‘new thinking’ on autonomy and accountability, the accountability demand is expressed as the formal demand that one give an account. There is no reduction of that account to economic concepts. One is free to offer whatever justification one sees fit. In other words, one is able to exercise one’s autonomy to respond to the generic accountability demand: account for yourself! Under this ‘new thinking’, then, accountability and autonomy are compatible.
Someone may be quick to point out that these are not really new definitions of the terms. So much the better!