Can we wear our failed studies like badges of honor?

I've been thinking about the brilliant and at times insidious human quality described in these two articles ("The Truth Wears Off", Dec. 13 New Yorker, and "Lies, Damned Lies, and Medical Science", November Atlantic) -- our remarkable problem-solving capacity. Given enough angles to view something, and enough time and incentive, we are very likely to stumble upon a seemingly impossible solution. The problem in scientific research is that it might be literally impossible, i.e., false. But we desperately want it-- the finding that will get us a postdoc, or a grant, or (gasp) an NPR interview!! These are the flights of fancy that entertain us at hour 12, 13, 14... as we stare at the same dataset, chewing on fingernails and pencils, cocking our heads, having moments of "insight" that lead to less and less well-informed strategies for MAKING THE THING WORK. Oh, data analysis, we say. You are a cruel mistress.

But finding the thing is not the problem, clearly. It's whether it's actually there to be found in the first place.

In the New Yorker article above, it is reported that Jonathan Schooler wants researchers to publish their proposals before beginning data collection, then track what happens as the study unfolds. This would bring some accountability to the process and maybe solve what I'm now calling "the problem-solving problem." The only trouble with this is it sets the bar a bit high. I'm in favor of making it incredibly, stupidly easy for researchers to publicly note things that didn't work. And to solidify their more exciting (i.e., positive) findings in the process. My idea is simply an index of websites (blogs, faculty webpages, etc.) where researchers are not tracking the ins and outs of every project, but simply letting each other and the public know which ones didn't quite work. Studies that do work would then get written up for publication as usual, but this gives us a way to see what the chaff looked like before it got blown disdainfully away from the wheat. Can you imagine how educational, how downright helpful that would be?

A likeminded approach that's gotten a bit of traction, which I applaud, is The Journal of Articles in Support of the Null Hypothesis. But to do justice to the vast number of failed studies there should be about ten such journals for every one that publishes positive results. Plus, journal articles take too long to write up, anyway, and who likes to linger on failure? Instead, let's get the word out quickly and succinctly, like so:

• What the idea was.
• How we operationalized and measured things.
• How big the sample size was.
• How near the miss was (p = .06?)
• Some very brief speculation about why it didn't work (totally optional, and not to be dwelled on).

When we start recording our failed studies, let's do so proudly. Why? Because every failed study gives credence to the ones that actually do work. The more failures we have, the less shady it will be when we find something real. Unfortunately our science news outlets (NPR!!) have been pretty dull to this up to now, and have not been nearly suspicious enough of the uncanny number of positive findings.

By publicly listing our failed studies, we build a strong case for our ongoing work. Who's with me on this? Can we wear our failed studies like badges of honor?

AT&T Anxiety Disorder (recommended for inclusion in DSM-V)

Diagnostic criteria for 300.xx AT&T Anxiety Disorder

A. Excessive anxiety and worry (apprehensive expectation), occurring more days than not for at least 1 month, about placing or receiving phone calls on an AT&T mobile device.

B. The person finds it difficult to control the worry.

C. The anxiety and worry are associated with four (or more) of the following eight symptoms:

(1) increased heart rate when mobile device rings
(2) constant, furtive glancing at device for evidence of signal
(3) excessive apologizing to person on other end of line
(4) frantic running in and out of doors in search of signal
(5) reducing complex ideas to short, clipped sentences, e.g., "I like Beth" as a substitute for "My feelings about Beth are that we could eventually be a good match but at the moment I am ambivalent, especially given her recent breakup with Tom"
(6) recursive phone conversations about unsuccessful phone conversations
(7) loss of train of thought between dropped calls
(8) resorting to email for all communications

D. The anxiety, worry, or physical symptoms cause clinically significant distress or impairment in social, occupational, or other important areas of functioning.

E. The disturbance is not due to the direct physiological effects of a substance (e.g., a drug of abuse, a medication) or a general medical condition (e.g., hyperthyroidism), or better accounted for by another disorder (e.g., Comcast Rage Disorder).

There, I fixed it

Why Andy Will Never Eat A Cow

Tonight I was talking to my roommate Andy in the kitchen and he told me an incredible story.

I was asking him how long he had been vegetarian, what his reasons were, etc. He made a curious comment: "I could never eat a cow. Mostly because of their remarkable sense of aesthetics." I must have looked puzzled. He explained with the following story.

"A few years ago, back in Texas, some friends of mine were recording an album in a little shack in the middle of a cattle ranch. I spent a couple weeks there hanging out and recording saxophone tracks for their record. We began to develop a real rapport with the cows. We would moo, and they would moo back. One night I went out to have a cigarette and I had my saxophone with me. I played one note, kind of a low note, and the three cows nearby immediately looked up and mooed back. So I played a little more and they kept mooing back. Then I started really playing-- like, a song. I looked up and saw 200 head of cattle stampeding toward me. When they got within ten feet they stopped and just stood there, transfixed. I ended up doing an entire 30-minute concert for them. Some of them started having sex while I was playing. To them, it was like a rock 'n' roll party. When I finally stopped they turned and walked away, back out into the field."

"And that's why I will never eat a cow again. I would rather eat a cat than a cow."

THE END