Can we wear our failed studies like badges of honor?

I've been thinking about the brilliant and at times insidious human quality described in these two articles ("The Truth Wears Off", Dec. 13 New Yorker, and "Lies, Damned Lies, and Medical Science", November Atlantic) -- our remarkable problem-solving capacity. Given enough angles to view something, and enough time and incentive, we are very likely to stumble upon a seemingly impossible solution. The problem in scientific research is that it might be literally impossible, i.e., false. But we desperately want it-- the finding that will get us a postdoc, or a grant, or (gasp) an NPR interview!! These are the flights of fancy that entertain us at hour 12, 13, 14... as we stare at the same dataset, chewing on fingernails and pencils, cocking our heads, having moments of "insight" that lead to less and less well-informed strategies for MAKING THE THING WORK. Oh, data analysis, we say. You are a cruel mistress.

But finding the thing is not the problem, clearly. It's whether it's actually there to be found in the first place.

In the New Yorker article above, it is reported that Jonathan Schooler wants researchers to publish their proposals before beginning data collection, then track what happens as the study unfolds. This would bring some accountability to the process and maybe solve what I'm now calling "the problem-solving problem." The only trouble with this is it sets the bar a bit high. I'm in favor of making it incredibly, stupidly easy for researchers to publicly note things that didn't work. And to solidify their more exciting (i.e., positive) findings in the process. My idea is simply an index of websites (blogs, faculty webpages, etc.) where researchers are not tracking the ins and outs of every project, but simply letting each other and the public know which ones didn't quite work. Studies that do work would then get written up for publication as usual, but this gives us a way to see what the chaff looked like before it got blown disdainfully away from the wheat. Can you imagine how educational, how downright helpful that would be?

A likeminded approach that's gotten a bit of traction, which I applaud, is The Journal of Articles in Support of the Null Hypothesis. But to do justice to the vast number of failed studies there should be about ten such journals for every one that publishes positive results. Plus, journal articles take too long to write up, anyway, and who likes to linger on failure? Instead, let's get the word out quickly and succinctly, like so:

• What the idea was.
• How we operationalized and measured things.
• How big the sample size was.
• How near the miss was (p = .06?)
• Some very brief speculation about why it didn't work (totally optional, and not to be dwelled on).

When we start recording our failed studies, let's do so proudly. Why? Because every failed study gives credence to the ones that actually do work. The more failures we have, the less shady it will be when we find something real. Unfortunately our science news outlets (NPR!!) have been pretty dull to this up to now, and have not been nearly suspicious enough of the uncanny number of positive findings.

By publicly listing our failed studies, we build a strong case for our ongoing work. Who's with me on this? Can we wear our failed studies like badges of honor?