Publishing the Wrong analysis

[fuzzylogician]

I wonder what you think about the following situation:

I presented a paper at a conference in November, and the written version for the proceedings is due next week. Since November I have discovered new data that can't be explained by the analysis I proposed back then. This new data causes a significant complication that is not predicted by any current theory that deals with the topic I am working on, and I haven't been able to fully solve it yet. Since the paper has already been accepted, my advisor suggested that a write a "progress report" of sorts, laying out the desiderata that a good theory should predict, and sketching my previous attempts to explain the data and the problems they raised. Now, I've always thought it's really wrong that researchers don't publish papers about failed attempts at solving a problem (with the explanation of why and what's missing for the analysis to work). What do you think - would you publish an analysis that only accounts for some of the facts and explain why it fails to explain the others? Or would you hold off and hope that you can produce a working analysis, and only publish that?

[pangur-ban]

It's just a proceedings, right, so it's not as serious a publication as a journal, eh? I'd go with what your advisor suggests for the proceedings and keep working on the problem. Then, once you have a working analysis, you could submit that to a journal (or another conference, or whatever) for publication.

Though I think, if it were me, I wouldn't want to publish an incomplete analysis. Still, I do like it when I find mentions in articles of problems for a theory or directions for future research.

[fuzzylogician]

It's true that it's only a proceedings, but as my advisor says, its a proceedings "that people actually read." So the paper will be out there and people will read it.

So far the pros I see to publishing the paper are: [a] an even more-wrong (wronger?) analysis is out there in the form of a handout from the conference talk, so it's not like no one knows what I'm working on or what I think (well, thought) about it; it's a paper from a good conference proceedings that goes on my CV, which is nice for a first-year; [c] it's putting a new idea and new data out there, which will people will be interested in.

There are also cons to publishing the paper: [a] it's incomplete; it's letting everyone know my new idea, before I had time to fully develop it. Someone could use it to publish a working solution of the problem before I get a chance to do it myself. [c] it's a huge strain on my time, and I'd have to write about my current analysis, which has problems, instead of spending the next 2-3 weeks[1] improving the analysis I have and making it work.

[1] the official submission deadline is this Monday. I am hoping for an extension.. otherwise this whole discussion will remain a theoretical one.

[pangur-ban]

I like "even more wronger" best. ("wronger" alone sounds weird to me)

I think fear of someone else getting there first would be my biggest concern in publishing the incomplete paper. But the other concerns are valid too. Still though, if your advisor isn't too worried about it, maybe it's fine?

I also doubt having the line on your CV is a huge concern -- you're giving another conference paper soon (or just recently?) too, right? If you get a proceedings out of that, then you're almost as well off as if you had this one. And you seem admirably productive in general, so I doubt that it will matter either way in a few years' time.

So I guess it comes down to (in my view) time/effort, and how much it means to you to get the data out and retracting (somewhat) what you said the handout. Or perhaps you can just let the decision be made by inaction, if you don't get the extension.

[fuzzylogician]

I like "even more wronger" best. ("wronger" alone sounds weird to me)

heh

I'm presenting basically the same stuff (more data, less analysis) in a workshop this Saturday. It's not prestigious at all and there's not going to be a proceedings. It's more a chance for me to practice public speaking, meet people and maybe get some new feedback on my analysis. I could in principle submit to another conference and follow up with a journal article anyway, assuming that I can actually work out something that accounts for all the data.

The decision that I recently discussed with my advisor is to proceed as if I am publishing - spend the time working out the details of the failing analysis, in part in order to diagnose the problem in full and in part to see how much I can convince myself that it's a big problem and not just a bump in the road. Right now it seems to me that the analysis is crashing in a way that I can't save, so I'll need to completely change directions. If that's the case, I'm not too worried about publishing it (though I could be wrong; I haven't had enough time to really think this through). Even if I work on this nonstop for 2-3 weeks *pleads to the gods of extensions* it won't be perfect and I'll eventually have to make a decision based on less than optimal information.

I'd love to hear more opinions about this situation.

Edited February 26, 2010 by fuzzylogician

[Genomic Repairman]

Is it going to be a short communications or a full on research article. If its a short communication I think just let it slide since those aren't definitive and just serve as springboards for ideas. I might try to think of something to address the holes in your hypothesis but go ahead and put it out, rarely if ever is there a complete story. Someone can always disprove you or you can always go back and disprove yourself when you have more evidence. I read proceedings, like PNAS, but I always read them with a grain of salt. Just because you put something out for public consumption does not mean that we take your or anyone's analysis as dogma. Remember be a skeptic.

[liszt85]

What do you think - would you publish an analysis that only accounts for some of the facts and explain why it fails to explain the others? Or would you hold off and hope that you can produce a working analysis, and only publish that?

I would publish the analysis and talk about the facts that it accounts for, and then would also give a sneak peak of the kind of data that it would not be able to capture (a brief mention in the conclusion section). Say that the research is moving in that direction in the conclusion section.. How big is this conference? Anybody likely to steal your idea (AND solve this quicker than you will)?

I would then shut myself up in a room for days/weeks and try to come up with a working solution ASAP and would type it up for publication as soon as I possible can.

[fuzzylogician]

Is it going to be a short communications or a full on research article. If its a short communication I think just let it slide since those aren't definitive and just serve as springboards for ideas. I might try to think of something to address the holes in your hypothesis but go ahead and put it out, rarely if ever is there a complete story. Someone can always disprove you or you can always go back and disprove yourself when you have more evidence. I read proceedings, like PNAS, but I always read them with a grain of salt. Just because you put something out for public consumption does not mean that we take your or anyone's analysis as dogma. Remember be a skeptic.

It has a 14-page limit, so it's not going to be able to cover everything. However, it's probably a bad idea not to address a problem, if I see it. Other people will surely raise it anyway.

It's not definitive even if it's published in the Journal of Truth. Someone will always challenge what you say, and that's a good thing.

I would publish the analysis and talk about the facts that it accounts for, and then would also give a sneak peak of the kind of data that it would not be able to capture (a brief mention in the conclusion section). Say that the research is moving in that direction in the conclusion section.. How big is this conference? Anybody likely to steal your idea (AND solve this quicker than you will)?

I would then shut myself up in a room for days/weeks and try to come up with a working solution ASAP and would type it up for publication as soon as I possible can.

I've been holed up in my office working on this every free minute I have for the last month, and I think it might time to take a step back and let it sit for a while before attacking the problem again and (hopefully) thinking of new ideas. The problem is not so much that I will have a "ideas for future research" section in the end of the paper as that my analysis gives a wrong prediction for the truth conditions of a certain kind of construction and it appears right now that that follows logically from what I propose so I need to completely drop that and do something else. It does give really nice predictions in other places and it's an interesting approach in general that several professors have told me that people will appreciate reading about. But yeah, I'll be publishing a "why what I say is wrong" paper.

[StrangeLight]

write the paper as you would have without the problematic data set, but shorten it by 4 or 5 pages. then insert a 4-5 page piece explaining the problem the new data presents near the beginning of the paper. you don't want to totally throw out the argument you had made at the conference but you can't pretend the data didn't happen. just write "more research needs to be done," and "this presents an interesting problem that merits deeper attention" a lot.