Negative Results – Not Just For Journal Articles?

David Bruggeman offers another twist on turning negatives into positives here. I’d like to add to this that it’s part of an ethos of not being afraid to make mistakes, even of valuing them. Some might refer to this as an entrepreneurial attitude.

David Bruggeman's avatarPasco Phronesis

There is a strong positive bias in how scientific knowledge is generated, written about, and measured.  It is easier to find research proving a hypothesis than replication studies that fail to confirm earlier findings.  It is easier to access explanations of why certain technologies came to be than studies about why we don’t have flying cars, or some other breakthrough promised to us through the magnificence of science and technology.  It’s an enormous hole in our understanding of the world, facilitated by the mores of the scientific reward system.

The same is true for metrics.  While the number of ways one can assess the impact of a particular paper is changing, many of the ‘alt’ metrics emerging are still thinking primarily in positive terms.  At least that’s the proposition of J. Britt Holbrook and some of his colleagues at the University of North Texas.  In a letter to Nature

View original post 185 more words

San Francisco Declaration on Research Assessment — Well done, DORA

Anyone interested in research assessment should read this with care.

DORA.

It’s been presented in the media as an insurrection against the use of the Journal Impact Factor — and the Declaration certainly does … ehr … declare that the JIF shouldn’t be used to assess individual researchers or individual research articles. But this soundbite shouldn’t be used to characterize the totality of DORA, which is much broader than that.

Honestly, it took me a few days to go read it. After all, it’s uncontroversial in my mind that the JIF shouldn’t be used in this way. So, an insurrection against it didn’t strike me as all that interesting. I’m all for the use of altmetrics and — obviously, given our recent Nature correspondence (free to read here) — other inventive ways to tell the story of our impact.

But, and I cannot stress this enough, everyone should give DORA a careful read. I’m against jumping uncritically on the bandwagon in favor of Openness in all its forms. But I could find little reason not to sign, and myriad reasons to do so.

Well done, DORA.

‘Pure hype of pure research helps no one’ says Sarewitz; what this says about freedom

I tend to agree with a lot of what Dan Sarewitz argues here:

Pure hype of pure research helps no one : Nature News & Comment.

But I also want to suggest that there’s an argument to be made against the High Quality Research Act that goes beyond Sarewitz’s claim that it helps no one.

To be fair, that’s just the headline. Sarewitz also claims something I think is a bit more controversial — that the HQRA is really nothing to get too worried about. Not only does it help no one, but also it doesn’t hurt anyone.

This strikes me as mistaken. I’ll try to articulate why in terms of the distinction between negative and positive freedom I’ve been exploring. Here goes.

First, I agree that the HQRA helps no one.  But it’s not just that the HQRA is redundant — though this is certainly true. It’s also that it doesn’t allow us to do anything more to demonstrate our accountability, as I think the Broader Impacts Criterion does. In other words, it doesn’t increase anyone’s positive freedom.

Second, it actually decreases our negative freedom. By requiring NSF to re-certify what the merit review process already certifies (at least when it’s working as designed), this ‘added layer of accountability’ actually just increases the kind of bureaucratic red tape we should be trying to decrease if we’re interested in an efficient government. This makes about as much sense as the Florida Blue Ribbon Task Force’s suggestion to charge more for classes in majors that supposedly won’t result in better jobs for graduates. Majors that result in higher paying jobs actually should be in greater demand, and so should cost more, not less. But not according to the Blue Ribbon Task Force (see pp 22-3).

Finally, I think the HQRA might be a case study in how to reconcile notions of positive and negative freedom — or at least how to think of both ideas of liberty as possibly working together. It’s sort of a test. Sometimes, a policy that might increase our positive freedom can be seen as decreasing our negative freedom. I think the NSF’s Broader Impacts Criterion is a case in point. Yes, it places an additional burden on researchers, and so in this sense it is a limitation on their negative freedom. But it also increases their positive freedom, so that a trade-off is possible. The HQRA, on the other hand, decreases our negative freedom without also increasing our positive freedom. Any policy that doesn’t increase either our positive or our negative freedom is highly questionable — or so I am suggesting.

Will The High Quality Research Act Diminish Our Collective Cognitive Dissonance?

I plan to follow up on David’s post and Dan’s argument in Nature, soon. Until then, enjoy this!

David Bruggeman's avatarPasco Phronesis

The High Quality Research Act is a draft bill from Representative Lamar Smith, Chair of the House Science, Space and Technology Committee.  Still not officially introduced, it has prompted a fair amount of teeth gnashing and garment rending over what it might mean.  The bill would require the Director of the National Science Foundation (NSF) to certify that the research it funds would: serve the national interests, be of the highest quality, and is not duplicative of other research projects being funded by the federal government.  The bill would also prompt a study to see how such requirements could be implemented in other federal science agencies.

There’s a lot there to explore, including how the bill fits into recent inquiries about specific research grants made by the National Institutes of Health (NIH) and the NSF.  (One nice place to check on this is the AmericanScience team blog.)

But…

View original post 278 more words

NSF Says No to Congressman’s Request for Reviewer Comments – ScienceInsider

The latest in the showdown between Rep. Lamar Smith and NSF.

NSF Says No to Congressman’s Request for Reviewer Comments – ScienceInsider.

Interesting to think about the limits of confidentiality here.

What does it take to be ‘liked’ by scientists?

Scientists don’t like me. Or, at least, they don’t show any evidence of liking what I have to say about NSF’s Broader Impacts Merit Review Criterion. Last week, I blogged this ScienceInsider interview (here and on the CSID blog) with an unnamed congressional aide connected with Rep. Lamar Smith and his efforts to add “an extra layer of accountability” to NSF’s Merit Review Process.

I also left a couple of comments in the comments section under the article itself. It’s possible for readers of ScienceInsider to press buttons to indicate their agreement — or not — with comments. The site then tracks the number of likes or dislikes (expressed by pressing up or down carrots), displays them with each comment, and moves those comments with the most likes up to the top.

Guess whose comments are dead last in line?

Here are the two most-liked comments:

lollardy3 days ago

Studying dairy production in China is a very poor choice for an example of what constitutes a bad grant. It has direct relevance to something most people in America consume every day. It could reduce cost for millions, increase food safety, improve the quality or nutrient density of a commonly consumed item, etc. Every time I hear a story on Fox about a “wasteful” study, I can usually think of ten ways it could benefit people and industry here. Somehow I think the time would be better spent putting in an “additional layer” to cover pentagon spending.

Kenneth DeBacker4 days ago

A lot of smoke is being blown by Rep. Lamar Smith’s aide. The aide’s answers are slick and cover’s the real intent of the bill- to politicize the sciences through selective funding or defunding of areas of study Republicans do not like. The most egregious example would be the ban on studying gun violence in America.

Each of them has received twelve likes.

I suppose if I were simply to say that Congress is out to politicize science or that Smith is out of his depth or that scientists should be left alone to pursue research however they wish, scientists might like that. But I’m willing to give Smith the benefit of the doubt, at this point. My contention is that he (or his aide) doesn’t yet understand the revisions to NSF’s Merit Review Process. If he did, then I think he’d see that accountability is already built into the process. I think Smith should not introduce the High Quality Research Act, but instead should seek to monitor how scientists respond to the new Broader Impacts Criterion.

But there’s a real problem with what I’m suggesting. And it’s not that Smith is a Republican out to get science. The problem is that scientists themselves don’t understand the Broader Impacts Criterion. They don’t understand that this is their last, best hope to preserve their academic autonomy while meeting accountability demands. And they don’t want to hear it, either.

To see my comments on the ScienceInsider interview, simply follow this link and scroll to the bottom of the page.

Pressure Builds on Congress to Kill NSF Bill – ScienceInsider

The letter from former NSF Directors and Chairs of NSB is on point:

The NSF, the National Science Board, and the Congress have regularly examined the merit review process and adjusted it, periodically, after widespread consultation with all parties concerned. One of the more recent changes was to elevate the importance of assessing the broader impacts of the proposed project on a par with assessing the scientific and technical merit. We believe this approach serves to strengthen the merit-based decision making process the Foundation uses for individual research projects.

Pressure Builds on Congress to Kill NSF Bill – ScienceInsider.

What Representative Lamar Smith Is Really Trying to Do at NSF – ScienceInsider

Fascinating. Comments are also worth reading.

What Representative Lamar Smith Is Really Trying to Do at NSF – ScienceInsider.

Communities of Integration Workshop – Field Philosophy | csid

Communities of Integration Workshop – Field Philosophy | csid.

Will be my first time to visit ASU. I’m excited to see it!

Autonomy, negative and positive freedom, and broader impacts

I’ve been continuing to think about the topic I broached in a previous post. There, I described what I termed ‘old’ and ‘new’ ways of thinking about autonomy and accountability. Of course, my ‘new’ ways are not all that new.

Here I want to sketch very briefly some recent thoughts I’ve had as a result of teaching my class this past semester at UNT in ‘Science, Technology, and Society’, my recent attendance at the ‘Broader Impacts Infrastructure Summit‘ at the University of Missouri, and recent discussions with colleagues. I want to thank my students for forcing me to try to get my thoughts in enough order to be able to stand up in front of them and teach. I also want to thank Steve Fuller for turning me on to some new reading material and Bob Frodeman and Adam Briggle, with whom I tend to discuss everything.

I don’t want to blame any of them for these thoughts, of course. But I mention them, in part, to indicate ways in which my own thinking is influenced by these ‘non-research’ activities. In particular, my teaching this past semester has hardly been separable from my research. This notion that teaching and research can be — and should be — mutually reinforcing in this way is one of two main Humboldtian theses about the university. For short, let’s call it the integration of research and teaching (though I’m on record about my suspicion of the term ‘integration’).

The other main Humboldtian thesis about the university is that it should be autonomous from the state. This same notion was expressed in the post WWII era by Vannevar Bush, whose “Science — The Endless Frontier” forms much of the conceptual basis of science policy. In my previous post on autonomy and accountability, I argued that we needed to get past an account of freedom as freedom from constraint by the state. What I did not say in the last post is that this is precisely the account of freedom that both Humboldt and Bush share.

This notion of freedom is very much like what Isaiah Berlin describes as ‘negative freedom’. Under such a negative account of freedom, any interference on the part of the state with the sort of research that’s performed at universities would be an assault on freedom. I think this is the sort of freedom most academics believe in, whether they know it or not. The idea is that academics should be free from outside interference. Only free in this sense will they be allowed to roam wherever their curiosity would take them.

According to my ‘new’ account of autonomy, however, freedom means self-determination rather than freedom from restraint. This notion actually corresponds in many ways to what Berlin describes in the essay cited above as ‘positive liberty’. Under this conception of freedom, requirements such as the NSF’s Broader Impacts Merit Review Criterion need not curtail academic freedom, as long as they don’t take away the academic’s ability to self-legislate. Instead, such requirements could be seen under the rubric of ‘interference without domination’ as outlined by Philip Pettit in his account of Republicanism (thanks to Steve Fuller for that reference — have been trying to read some about Pettit and Republican political theory in the past day or two).

I am not quite comfortable with transferring ideas straight from political philosophy to the realm of knowledge policy. But I’m not worried here about making a mistake in the details. I think there’s something helpful in seeing things this way.

One more point that may seem unrelated — though it isn’t, I just won’t be able to make all the connections here. At the Broader Impacts Summit, it became clear that there were two ‘camps’ forming. Again, I’m going to speak loosely. One camp wanted to claim broader impacts as its area of expertise — let’s call them the BI-Experts, though I believe most of them would consider themselves experts in science education and outreach. What they were pushing, albeit in still-developing form, was to make broader impacts activities a side show to the real research. It would be a rigorous sideshow, mind you. But broader impacts activities ought to be the realm of BI-Experts, rather than scientists, who can’t tell outreach from a hole in the ground.

The other group — and interestingly, it tended to include former and current NSF program officers — had what I labeled a Humboldtian view of broader impacts. The idea was that intellectual merit and broader impacts should be integrated in the project (and perhaps in the person of the Principal Investigator). Without wanting to limit broader impacts to teaching, this view is consistent with the Humboldtian position on the integration of research and teaching.

However, the Humboldtian take on broader impacts actually depends on a positive notion of freedom. If one were to take a negative view of freedom (as I think Humboldt does), then one would be more inclined toward the BI-Expert approach. Why? The BI-Experts will run interference for the scientists, who will then be free to perform their research without having to be concerned with broader impacts.