Advisor Picked Method We Didn't Really Understand - Presenting Work I Know Is Bad?

[MistakesWereMade]

My advisor assigned me a project using a statistical method I don't think he fully understood. It's new, one he hasn't used before, and it is in a dense, poorly written article. I have expressed to him that I felt we didn't understand it well enough, that we should consider collaborating with others to make sure the method was tenable, and that I felt it might not work. However, he blew these concerns off. When the initial results looked promising, I let it go.

A few months later, my latest results look weird as hell. I reread the article for hundredth time to see if we had missed something. I think I found it. I have good reason to believe that our data violate a key assumption of the method (not described by the authors!) and that our results are complete garbage.

The problem is that I am signed up to present these results as a poster in a few months time at a conference to which I have already been accepted and funded to attend. I don't know how I can back out with my plane and hotel already bought by the department. What do I do? Keep mum? It's wrong and I'm worried someone will find me out anyway. 

I have emailed my advisor, but he is on vacation and hasn't responded yet. 

Please proceed from the assumption that I am correct. I don't need to troubelshoot a scenario where everything is A OK. I need to troubleshoot the potentially terrible mistake I have made. Even though everyone in the field knows that beginning students don't design these projects, my head is on the chopping block; I am ultimately responsible for the research I put my name on. 

Knowing what I know now, I won't let it get to the publishing phase before the issues I identified are resolved, but what do I do about the conference?

 

Edited May 29, 2016 by MistakesWereMade

[Concordia]

Can the analysis be re-run correctly?  Or will that take too long?  Or (if I understand 'violate a key assumption of the method' correctly, do you simply have the wrong batch of numbers to be doing any analysis on?

Anyway, even if the department has reserved the hotel, etc., surely that is refundable, or at least transferable.  It would look terrible for them to send you out to present an illogical and completely incorrect paper.

[MistakesWereMade]

20 minutes ago, Concordia said:

Can the analysis be re-run correctly?  Or will that take too long?  Or (if I understand 'violate a key assumption of the method' correctly, do you simply have the wrong batch of numbers to be doing any analysis on?

Anyway, even if the department has reserved the hotel, etc., surely that is refundable, or at least transferable.  It would look terrible for them to send you out to present an illogical and completely incorrect paper.

wrong batch of numbers to do any of the analysis, unfortunately :-(

 

Oh, and it's a poster presentation, but still not great. I am editing my post to specify a poster now. How I worded it, it's not really clear.

I also can't afford to pay for the plane and hotel if they don't refund me. I haven't heard of anyone doing something like that.

If my advisor doesn't reply soon with some kind of solution, I'm honestly consider faking illness or something. I don't know what else to do. 

Edited May 29, 2016 by MistakesWereMade

[fuzzylogician]

I don't think there is a reason to be in panic mode just yet. The conference is a few months out. That should be more than enough time to figure out if there are mistakes and to either find a way to fix them or withdraw from the conference. You should not present something that you do not want to put your name on. I may be reading between the lines here, but it sounds to me like you worry that your advisor will dismiss your concerns. I hope that you are wrong and your advisor takes you seriously and can either show you that there is nothing to worry about, or fix whatever needs fixing (or scrap the project if the mistake makes it a non-starter). If your advisor insists on continuing and you no longer believe in the validity of the results, then you may not have a choice except to ask to remove your name from the work; it may not be your choice what happens to the project. But this all sounds a little premature, given that there is still a lot of time to deal with this, and you haven't heard back from your advisor yet. Since he's on vacation, I think he is within his rights not to respond to email and not to try to understand your concerns just now. When he's back, that's when you should talk to him.  

That said, (to address something else in your post) I don't think that presenting something that is ultimately found to be wrong is inherently bad -- everyone makes mistakes or works off of limited and imperfect data/assumptions, and that is how science progresses. You argue against others all the time and that doesn't mean that you don't respect their work, just because you have a better method or new way to interpret data or new data that leads to a different conclusion. Best that you can do is argue what you believe to be correct based on the evidence in front of you at the time you make the argument. The crucial bit is being convinced you are correct when you make the argument, based on a good faith effort to interpret the data, even if later something changes. Of course that is different presenting something you believe to be wrong, which I don't think you should do.

[MistakesWereMade]

17 minutes ago, fuzzylogician said:

I don't think there is a reason to be in panic mode just yet. The conference is a few months out. That should be more than enough time to figure out if there are mistakes and to either find a way to fix them or withdraw from the conference. You should not present something that you do not want to put your name on. I may be reading between the lines here, but it sounds to me like you worry that your advisor will dismiss your concerns. I hope that you are wrong and your advisor takes you seriously and can either show you that there is nothing to worry about, or fix whatever needs fixing (or scrap the project if the mistake makes it a non-starter). If your advisor insists on continuing and you no longer believe in the validity of the results, then you may not have a choice except to ask to remove your name from the work; it may not be your choice what happens to the project. But this all sounds a little premature, given that there is still a lot of time to deal with this, and you haven't heard back from your advisor yet. Since he's on vacation, I think he is within his rights not to respond to email and not to try to understand your concerns just now. When he's back, that's when you should talk to him.  

That said, (to address something else in your post) I don't think that presenting something that is ultimately found to be wrong is inherently bad -- everyone makes mistakes or works off of limited and imperfect data/assumptions, and that is how science progresses. You argue against others all the time and that doesn't mean that you don't respect their work, just because you have a better method or new way to interpret data or new data that leads to a different conclusion. Best that you can do is argue what you believe to be correct based on the evidence in front of you at the time you make the argument. The crucial bit is being convinced you are correct when you make the argument, based on a good faith effort to interpret the data, even if later something changes. Of course that is different presenting something you believe to be wrong, which I don't think you should do.

I'm not mad at my advisor for not responding to my email; I just sent it anyway. 

If I am right,and I'm 85% sure I am, there is no redeeming this project though. It could potentially be reworked after the conference if we were able to access additional data (this is all secondary data analysis), but the only sample I think would be suitable is under a publisher's embargo until fall sometime. 

As for my advisor dismissing my concerns, I don't think he will do so after seeing what I sent him and maybe re-running my code himself for good measure. I am more worried about him being angry that I did not catch this earlier. I myself feel like I should have caught this earlier considering how many times I have read the paper the method is in, as poorly written as it is. 

In addition to this, I have found another problem with the method that would prevent us from doing something else with the data my advisor planned to do. While it does not completely destroy the project like the other problem, it is such a simple thing that I really have no excuse to have missed it. I didn't email this tidbit to him yet because I didn't want to send two freak out items in one email. 

[fuzzylogician]

You need to schedule a meeting with him when he returns from his vacation and go over both your concerns with him in detail. I can't decide if you think he'll just continue despite being proven wrong or that he'll just get mad at you. The fact that he doesn't take your word that there is a serious problem that should cause him to pull the plug on the project because of an email doesn't sound surprising to me, the question is how seriously he takes it and how he handles it when he's back. You don't say how far along you are in your program but it seems to me that the best thing you can do is go over the details to convince yourself you're right and then talk to him and take it from there. If he is upset that a lot of time and effort have been wasted because of this mistake, he may have a valid point and you'll just have to deal with it. It could be a good opportunity to rethink your process to see if you can understand why it took so long to discover the mistake and if there is anything you could have done differently that would have led to a better outcome. If he isn't convinced and can show you why you're wrong and there is really no problem, again it is a learning opportunity. And if you can't become convinced that the project can proceed but your advisor still wants to go ahead, that's when you have a decision to make, but again, I think that's a ways away from a real concern right now. 

[St Andrews Lynx]

Depending on how the conference abstract/title is phrased, you perhaps have the ability to present something else at the conference. In the upcoming months you could run a slightly different analysis and present the new results instead of the controversial data. The great thing about poster presentations is that it gives you an opportunity to interact with experts who can give you new ideas or advice about how to expand/improve your project.

When preparing for meeting with your advisor, try to be as proactive as possible. Think about various alternative plans and bring those to the table. Assume that your PI is not going to tell you to just give up entirely on the project. How can you rework or re-run the data? Is there a better statistical model you could apply? What are the short-term fixes? What are the long-term fixes? If your only contribution to the discussion is "It's Wrong! It's Wrong! It's All Wrong! Everything Is Hopeless!" then I could imagine your PI getting annoyed. Certainly don't blame the PI for any of this ("You didn't read through that paper properly"/"You dismissed my concerns out of hand."). Apologise briefly for not realising the problem(s) sooner, but 1 short apology early on in the discussion should be enough. 

 

[juilletmercredi]

^This was going to be my suggestion. If there's no way to salvage the exact project you proposed, you can usually substitute in a closely related set of analyses and present that instead. I see people at conferences with different titles and projects on their posters than are in the abstracts all the time! It's sort of an occupational hazard of requiring abstracts 6 months in advance when - face it - most people have only run cursory preliminary analyses by then. You may have to run a simpler set of analyses than initially promised, but you don't want to go in with completely wrong analyses. There's always a (decent!) chance that someone who knows the method well visits your paper out of curiosity and notices the error.

That's not the end of the world either - conferences are supposed to be about getting feedback on in-progress work, and that would be a useful piece of feedback if you didn't already know there was an error.

By the way, don't beat yourself up about this. Advanced statistical methods was one of my concentrations in graduate school. Often the only way to really know a new method is to just get in there, roll up your sleeves and try it out. Blundering around with it and making mistakes is completely normal: you are going to screw up the code, violate assumptions, get weird results and tweak and tune the model until one day it clicks and you get it right. That's what the process of learning brand-new methods that no one is around to teach you is about. When I was writing my dissertation I spent at least 3-4 months trying to get the dang model I built to run and fixing it so it made any kind of sense. I pored over lots of theoretical articles to fix little aspects of the data and the model itself to make it work. It's fun! Or it should be. Don't be afraid to make mistakes; mistakes are part of the discovery and learning process to success! And when you make them, you learn what to avoid later for round #2.

There's nothing inherently wrong with planning to use statistical methods you don't 100% understand, as long as you set out to understand the method in the process of using it. (And even then, you may never 100% understand the underlying theoretical/mathematical base of the model, but nobody expects that either unless you're a statistician or mathematician.)

Also, I don't know what field you are in, but in my field everyone does not assume that someone other than the student designed the project. I think most researchers understand that beginning students didn't design the entire larger project from beginning to end, but the understanding is that even first-year students often design smaller experiments within the project or take a chunk of the data and independently decide on a research question and plan of analysis to pursue.