Tuesday, January 15, 2008

When you screw up

As a scientist, the onus lies on you: to be your own critic, to check your work backwards and forwards before you report it to the community; because your word (when properly supported, of course) will be taken at face value. There is no policing at the level of coming into your lab and checking if you really mixed tube A with tube B and got C: if you say so and you have supporting data, it will be accepted after critical evaluation, which may not necessarily catch that you in fact mis-labeled tube A as tube B instead. Which is why one repeats experiments and adheres to an absolute code of conduct. It is also why I believe there should be some level of internal policing in labs: not because of lack of trust, but to ensure that one did not, as an honest mistake, mis-label and report erroneous results. Sometimes it takes a third eye to catch that. It would be ideal if all the raw data was examined by someone other than yourself. But ideal is far from real: real world has deadlines, busy everybody, races to publish before competition, and little need to show raw data.

That said, I think most of us have made mistakes at some time or the other. What happens then? I remember two occasions in my grad-school life where I committed big mistakes but luckily caught them in the nick of time.

The first incident was when I was collaborating with another prof on his work. He approached me at a meeting and discussed some data with me that he wanted me to look at, with the carrot of authorship attached. I thought I did a thorough job but a small human error crept in somewhere and slipped through the cracks. My own advisor (lets call her PhdAdv) had no role to play in this collaboration, hence was not even looking over my shoulder. Collaborator already considered me the "Expert", so took whatever I said without questioning. Figures were made and the paper was ready to go out the door. Collaborator had heaped lots of praise on me for my hard work and sent me the final version of the manuscript to go over. Thats when I discovered that I had made a huge mistake. (Literally mis-labeled file A as file B and run the analyses accordingly). I was mortified. The very thought that I had "almost" published something wrong was killing. Then, there was the issue of admitting to such a careless, avoidable mistake. This "wrong" analysis of mine gave rise to two big figures in the paper, that Collaborator's grad student really needed to graduate. Morever, I really liked Collaborator. I had dreams of doing a post-doc with him. This was a very early stage in my grad school and I could see my whole reputation being tarnished right there. I felt like shit.

I wrote to Collaborator about it, profusely apologizing backwards and forwards. He wrote me back the nicest email ever. First, appreciated the fact that I had caught it and owned up. Second, told me clearly that I should re-do the analyses, but if I didn't find anything interesting my section would be taken out and I would of course, have to forgo authorship. Third, that this incident did not change his impression of me at all and that he looked forward to collaborate on future projects with me. I re-did the analyses, the new pictures were not as pretty but still went on the paper. We collaborated again a couple years later and he had a standing offer of a position in his lab for me when I was close to graduating. I ended up not going but that's a whole other story.

The second time round, the mistake I made was again at a critical point. This time I had come up with a very nifty script to analyse my data. Or so I thought. Of course I subjected the script to lots of tests and it passed all of them. I even had my programmer lab-mate look over my script, and she okayed it. I was feeling very good about myself and put all my data sets through the script and generated tons of numbers. More analysis later, I came up with a result that looked like this


It was beautiful! An additional piece of data that fitted my hypothesis. And it was done two days before I was going out of the country to give my first ever talk at an international meeting. I showed it to PhdAdv and she was very kicked. She asked me if I had double-checked my script, and I showed her the test runs and their results. I was super-excited. She was satisfied. We incorporated the figure in my talk: it was really the crowning glory to my talk. That day I gave a practice talk to the lab, stayed back late finishing up experiments and went home close to midnight. It also happened to be the night Candy died on me, right in the middle of the road. It was way past mid night by the time I reached home after getting the car towed and everything. I sat staring at my data, and suddenly a small doubt occurred to me. I ran my script through some more tests. It failed them. I emailed the script to my programmer friends, explaining to them what it intended to do. S wrote me back right away "is mein bug hain". My world came crashing on to me. And now, I had to tell PhdAdv about it! It was literally my worst nightmare come true. I stayed up that night and redid the analyses after S had fixed my code. The new, correct picture looked like this


Not so beautiful any more. In fact not anything. I called up PhdAdv first thing in the morning. She was of course, angry. "I asked you if you had double-checked your script" she said. We discussed the new results, cross-checked before including them, deleted the old slide and toned down the whole presentation. PhdAdv made me give one more practice talk, whipping me into shape and restoring my confidence. I got on my flight. Gave my talk at the meeting. It was a super-hit. I got one nasty questioner, a huge debate followed and I could see all the big wigs in my field jump to my defense to shut up the obnoxious guy. I held my own too. The debate and discussion over my talk overflowed at the post-session coffee time. Nobody had to know that I had goofed up and almost presented a wrong result. I may not have had the most overwhelming or mind-blowing results then, but my methods and findings still showed a lot of promise that went down very well with the audience. I shudder to think what would have happened if I had shown the wrong figure, woo-ed the audience, only to realise later that i had actually made a big mistake and produced misleading results.

Yes, I learned my lessons to be careful, that one can never be too sure while cross checking, running controls, repeating experiments, duplicating results. It is very critical in science. The temptation to see what you want to see looms large and can sometimes unknowingly mar your view. Haste is also not a good idea: its important to put time and distance between yourself and your data so you can look at it objectively. But the most important thing of all, is that if you do screw up, to never be afraid to own up, no matter at what stage you discover your mistake. You owe it to the scientific community to correct yourself: to convey the right information and prevent others from wasting their time trying to reproduce your work. And in spite of the deep sense of shame and drop in confidence that overwhelms you when you first discover your mistake, you will sleep well at night after making repairs.

28 comments:

sd said...

Very very good post!

I guess everyone has a similar story. In my Masters thesis, I thought I had found an algorithm that beat an algorithm published in PNAS hands down [incidently this algorithm classifed genes...] After I had defended succesfully and was making some "minor corrections" I discovered my experiments were different from the PNAS paper's experiments.....thankfully the error did not creap into my thesis!

I request you to not delete this post [may be highlight it in your archives?]. I may need to come back to it some other day to draw inspiration and strenght to own up a mistake!

Janefield said...

great post for newbies/students in your field. definitely a stickie-for others now that you have learnt the lesson well and don't need to remind yrself.

deserves to go into SC blog and even college journals/newsletters in your field.

lovely! for the non science person also there is enough common sense advise to take from this piece. thank you :)

shub said...

Like Jane above said, "for the non science person also there is enough common sense advise to take from this piece." So true!

Easily one of the best and most honest posts I've read in a long, long time. Kudos, tgfi, you're an inspiration :)

hugs!

The_Girl_From_Ipanema said...

sd,
[incidently this algorithm classifed genes

heh. may be we could collaborate some day. :)

it seems like your kind of mistake would've at least been caught by reviewer, if you directly compared your algorithm over the published one. no?

jane,
yeah, i guess it applies in all walks of life. one sometimes sees retractions in scientific journals too: for the same reason. It's almost heartening in some ways to know that people come back to clarify their mistakes.

okay guys, will add this post to my list of stickies. it happened a long while ago and i'm over the sense of shame i felt then.

shub,
:) seriously, no. i just wrote in public domain how i badly screwed up. twice. as a scientist, there's no excuse for it. :/
but this discussion recently came up in my new lab and i wanted to record my experiences for myself.

the ghost of poor neihal who has been banned for no good reason , but who was so moved by this ..... said...

Lovely post. I was going to skip this one after first line thinking it was some parasite post, but I am glad I read through. As a non science person it just made me realise how much effort goes into all those papers and researches that we see and read about all the time never bother ourselves with. Very well said!

*post that she decided to comment nonetheless.

PS: And incase you didnt get that I was also trying to see the max.charterer limit for Nicknames and now I know. :D
Yeah yeah, I have waaaaaaaaay too much time on my hands.
So what!!!

The_Girl_From_Ipanema said...

@ t.g.o.n....

lol @ nickname

I was going to skip this one after first line thinking it was some parasite post

:@ GHOST OF NEIHAL, YOU ARE BANNED TOO!

shub said...

either way, you've moved up quiiite a few notches in my books :)

La vida Loca said...

Hey nicely written and well said

Sakshi said...

@TGFI - Oh, similar story. Will write later. Now need coffee!

ferret said...

awesome post,, made me realise that i've never made any mistakes :P

sd said...

"it seems like your kind of mistake would've at least been caught by reviewer, if you directly compared your algorithm over the published one. no?"

You are right. But, if I can help it, I would like to find all my mistakes at least in methodology, proofs etc [compared to a disagreement with the referee in perspective, conclusion, or a point of view]- I guess it all comes down to the fear that you spoke about in your post.

sd said...

I realize that you were talking about errors that *may not have been discovered by the refrees*. But some day some one will try and repeat an experiment/read through a proof more dilligently...

An interesting anecdote...on the other extreme: Reimann, the great
mathematician was extremely careful with what he published. As many people know, he has this famous conjecture on number theory in one of his papers. The Reimann conjecture (hypothesis) remains open to this date. However, he ofcourse knew much more then he wrote in the paper since he was super careful. The story goes that after hundred years of his death some one found his rough sheets and gained deep understanding of this conjecture, an understanding which had eluded mankind for those hundred years!

"heh. may be we could collaborate some day. :)"

Sure! Lage haath eke NSF grant proposal bhi likh dete hain:P

Apologize for these long comments, I am feeling quite jobless... Actually I am trying to write up something and lethargy has struck....

Sunil said...

TGFI....this is a really good post. I think almost everyone in research has had times when they made mistakes, or *almost* published a mistake and caught it in time. But it is almost impossible to understate how important it is to own up to your mistakes if you find you've made one.

Here's another situation (going off on a slight tangent). Let us say you've made a mistake, and you've published a paper with the mistake in it. Now, the mistake is relatively minor, and doesn't necessarily affect the conclusion of the paper. But it *might* alter the implications of the work, and more importantly, might have been an important figure that allowed the paper to be published in that journal. What do you do now? Retract the paper, and mostly good work? Or write to the editor? Or do nothing?

I actually really like the ethics classes/case studies we are often required to take in most academic institutes. There are the clear black and white cases, and then so many cases of different shades of gray in between.

ggop said...

I love these vignettes into research life. You displayed intellectual honesty in the face of big deadlines. What a pleasant contrast to stories of fudged research!

Tabula Rasa said...

great post!

some time back i wrote about the converse of this - finding patterns where none were expected.

The_Girl_From_Ipanema said...

shub,
:)

vida loca,
thank you. :)

sakshi,
put post. methinks we can make this into a tag of some sort. :p

ferret,
ahahahhahahaha! i am glad it took a post on my screw ups for you to realise that. :)

sd,
yeah, i know what you mean. its embarrassing enough to have to admit ones mistakes to a P.I or a collaborator. Its even worse to admit it to a review committee.

wow. that was a neat story about Reimann. Keep em coming. I like!

lol @ NSF proposal. :)
no need to apologize yaar. i totally feel your pain about writing and lethargy. sigh.


sunil,
Let us say you've made a mistake, and you've published a paper with the mistake in it.

i would say definitely write to the editor. then let him and the reviewers decide. if it went into a figure, its doubly important to rectify, even if it doesn't alter the final conclusion. Integrity is one thing, and the fact that there might be some poor sod somewhere trying to reproduce it...
Do nothing is definitely not an option. How would you sleep? (:))

Yeah, I quite enjoyed those ethics classes too (mostly because they were easy As, though. :p), and you are right, there's lots of grey...

ggop,
:) You displayed intellectual honesty in the face of big deadlines
at the risk of sounding terribly annoyingly self-righteous and pompous, and i'm really really sorry i dont mean to sound like that, but the truth is, i didn't really think there was a choice.

professor TR,
thanks and the converse you say? i am curious. will stop by to read it after chai has been had, thanks for the link.

lekhni said...

Very good post. it's tough to own up mistakes, or talk about them later. And though everyone makes mistakes, the higher up one goes, the more one pretends that he/she is perfect :) I'm not sure of academia, but this is definitely true in corporatedom..

Coco Captive said...

Awesome... it takes a lot to walk up to your superior and take accountability to your mistake.
This post should be bookmarked in the sidebar, Jallo!

Neihal said...

:D
how come you miss all the good part..like "lovely post"....and all that....and focus on things like "feeni, parasite posts"...I think you are nitpicking....I think you have some bigger purpose in banning me from this place... but that is not fair.

I DEMAND JUSTICE FOR NEIHAL AND HER GHOST !!!

Priyanka said...

That was one well written, interesting and informative post! :)

Anonymous said...

What a nice blog you have! Its really charming.

I felt like having an elder sister like you :-)

The_Girl_From_Ipanema said...

lekhni,
thanks. yes, it's tough to own up. but if you've to choose between owning up and bearing that shame, or putting wrong information out there to be set in stone, it puts things a bit in perspective, i think. and oh, academia has its own share of egotistic people, its no hallowed portal as they would like others to believe.

cc,
yeah. see my reply to lekhni. disseminating wrong information is not an option. hence.

neihal,
*rolls eyes*
she is back again.

priyanka,
thank you. :)

anony,
AWWW! given that my elder sister is one of the best things in my life, your comment has made my day! :D. You are awarded "best comment on this blog award" :p

Anonymous said...

tgfi,
your comment made my face look like this.

~A

The_Girl_From_Ipanema said...

A,
:-)

Tachyoson said...

oh boy, crosschecking and checking again is so VERY important. that i cant nearly stress it enough.

but trying to be 100% correct 100% of the time is not achievable ... its good when you get the chance to clarify and the other person doesnt show you down.

we're all human, after all.

The_Girl_From_Ipanema said...

tachy,
yeah, indeed. we're all human after all. :)

Diabolical Devil said...

Hmm so true.., imagine being at the other end and have your own "korean stem cell story"

The_Girl_From_Ipanema said...

hi diabolical devil,
yeah. scary prospect that.