This short post from Andy Revkin combines several of my interests: fracking, peer review, and scholarly communication.
This paper describes the results of a survey regarding the incorporation of societal impacts considerations into the peer review of grant proposals submitted to public science funding bodies. The survey investigated perceptions regarding the use of scientific peers to judge not only the intrinsic scientific value of proposed research, but also its instrumental value to society. Members of the scientific community have expressed – some more stridently than others – resistance to the use of such societal impact considerations. We sought to understand why. Results of the survey suggest that such resistance may be due to a lack of desire rather than a lack of confidence where judging impacts is concerned. In other words, it may be less that scientists feel unable to judge broader societal impacts and more that they are unwilling to do so.
Absent from the many analyses and discussions of scientific peer review are two intangible but very important byproducts: 1) feedback to the applicant and 2) exposure of the reviewers to new hypotheses, techniques, and approaches. Both of these phenomena have a virtual mentoring effect that helps move science forward. Such learning can occur as a consequence of both manuscript review and grant application review, but the review of grant applications, by its very nature, is more iterative and impacts the direction in which research moves very early in the investigation.
There are at least two funding agencies that recognize this phenomenon in the actual design of their peer review processes, so they deserve mention. The idea is to include non-academics as peer reviewers precisely to effect the sort of co-production of knowledge the article above suggests.
The second is the US Congressionally Directed Medical Research Program. Details of their peer review process are available on their website here.
Scientists don’t like me. Or, at least, they don’t show any evidence of liking what I have to say about NSF’s Broader Impacts Merit Review Criterion. Last week, I blogged this ScienceInsider interview (here and on the CSID blog) with an unnamed congressional aide connected with Rep. Lamar Smith and his efforts to add “an extra layer of accountability” to NSF’s Merit Review Process.
I also left a couple of comments in the comments section under the article itself. It’s possible for readers of ScienceInsider to press buttons to indicate their agreement — or not — with comments. The site then tracks the number of likes or dislikes (expressed by pressing up or down carrots), displays them with each comment, and moves those comments with the most likes up to the top.
Guess whose comments are dead last in line?
Here are the two most-liked comments:
Studying dairy production in China is a very poor choice for an example of what constitutes a bad grant. It has direct relevance to something most people in America consume every day. It could reduce cost for millions, increase food safety, improve the quality or nutrient density of a commonly consumed item, etc. Every time I hear a story on Fox about a “wasteful” study, I can usually think of ten ways it could benefit people and industry here. Somehow I think the time would be better spent putting in an “additional layer” to cover pentagon spending.
A lot of smoke is being blown by Rep. Lamar Smith’s aide. The aide’s answers are slick and cover’s the real intent of the bill- to politicize the sciences through selective funding or defunding of areas of study Republicans do not like. The most egregious example would be the ban on studying gun violence in America.
Each of them has received twelve likes.
I suppose if I were simply to say that Congress is out to politicize science or that Smith is out of his depth or that scientists should be left alone to pursue research however they wish, scientists might like that. But I’m willing to give Smith the benefit of the doubt, at this point. My contention is that he (or his aide) doesn’t yet understand the revisions to NSF’s Merit Review Process. If he did, then I think he’d see that accountability is already built into the process. I think Smith should not introduce the High Quality Research Act, but instead should seek to monitor how scientists respond to the new Broader Impacts Criterion.
But there’s a real problem with what I’m suggesting. And it’s not that Smith is a Republican out to get science. The problem is that scientists themselves don’t understand the Broader Impacts Criterion. They don’t understand that this is their last, best hope to preserve their academic autonomy while meeting accountability demands. And they don’t want to hear it, either.
To see my comments on the ScienceInsider interview, simply follow this link and scroll to the bottom of the page.
ScienceInsider‘s take on Obama’s speech at the National Academy of Sciences: