Tag: arsenic

Stop deifying "peer review" of journal publications:

Peer review.  It is a critical part of scientific research and scientific progress.  Without it, science as a field might look like Fox News Stories or postings on Jenny McCarthy’s web site, where ideas people have are given gravitas regardless of how ludicrous they are.  But somehow, many in the public and press, and many many scientists alas, have deep misconceptions about peer review.

The most recent example of such misconceptions involves the arsenic life saga.  If you are not familiar with this story – here is a summary (for some fine scale details on the early parts of the story see Carl Zimmer’s post here).

In November 2010 NASA announced that in a few days they would hold a press conference discussing a major finding about life in the universe.  On December 2, 2010, they held their press conference and discussed a paper that was in press in Science from multiple NASA funded authors including Felisa Wolfe-Simon.  The paper was of interest because it claimed to have shown that a bacterium was able to replace phosphate in its macromolecules, including its DNA, with arsenic.  The press conference made claims that were very grandiose, like that textbooks would have to be rewritten, and the study of life on Earth and elsewhere would have to be completely rethought.

After a few days of mostly very glowing press reports, a few critiques began to emerge including in particular one from Rosie Redfield, a microbiologist at the University of British Columbia.  The critiques then snowballed and snowballed and the general consensus of comments appeared to be that the paper had fundamental flaws.  Some of the critiques got way too personal in my opinion and I begged everyone to focus on the science not personal critiques.  This seemed to work a little bit and we could focus on the science, which still seemed to be dubious.  And many, including myself, expressed the opinion that the claims made by the authors in the paper and by the authors and NASA in the press conference and in comments to the press, were misleading at best.

Now critiques about new findings are not unusual.  We will get back to that in a minute.  But what was astonishing to me and many others, was how NASA and the authors responded.  They said things like:

… we hope to see this work published in a peer-reviewed journal, as this is how science best proceeds.

and

It is one thing for scientists to “argue” collegially in the public media about diverse details of established notions, their own opinions, policy matters related to health/environment/science. 

But when the scientists involved in a research finding published in scientific journal use the media to debate the questions or comments of others, they have crossed a sacred boundary [via Carl Zimmer]

and the kicker for me was a letter Zimmer posted

Mr. Zimmer, 

I am aware that Dr. Ronald Oremland has replied to your inquiry. I am in full and complete agreement with Dr. Oremland’s position (and the content of his statements) and suggest that you honor the way scientific work must be conducted. 

Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated. You can see many examples in the journals Science and Nature, the former being where our paper was published. This is a common practice not new to the scientific community. The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner. 

Regards,
Felisa

This was amazing since, well, they were the ones who held the overhyped press conference.  And then I (and others) found it appalling that they in essence would not response to critiques because they were not “peer reviewed.” I told Zimmer

Whether they were right or not in their claims, they are now hypocritical if they say that the only response should be in the scientific literature.

Zimmer had a strong defense of scientists “discussing” the paper:

Of course, as I and others have reported, the authors of the new paper claim that all this is entirely inappropriate. They say this conversation should all be limited to peer-reviewed journals. I don’t agree. These were all on-the-record comments from experts who read the paper, which I solicited for a news article. So they’re legit in every sense of the word. Who knows–they might even help inform peer-reviewed science that comes out later on.

(I note – yes I am quoting a lot from Zimmer’s articles on the matter and there are dozens if not hundreds of others – apologies to those out there who I am not referencing – will try to dig in and add other references later if possible).

And so the saga continued.  Rosie Redfield began to do experiments to test some of the work reported in the paper.  Many critiques of the original paper were published.  The actual paper finally came out.  And many went about their daily lives (I keep thinking of the Lord of the Rings whisper “History became legend. Legend became myth. And for two and a half thousand years, the ring passed out of all knowledge.”  Alas, the arsenic story did not go away.

And now skipping over about a year.  The arsenic story came back into our consciousness thanks to the continued work of Rosie Redfield.  And amazingly and sadly, Wolfe-Simon’s response to Rosie’s work included a claim that they never said that arsenic was incorporate into the bacterium’s DNA.  (I have posted a detailed refutation of this new “not in DNA” comment here).

But that is not what I am writing about here.  What is also sad to me are the continued statements by the paper’s authors that they will not discuss any critiques or work of others unless they are published in a peer reviewed article.

For example, see Elizabeth Pannisi’s article in Science:

But Wolfe-Simon and her colleagues say the work on arsenic-based life is just beginning. They told ScienceInsider that they will not comment on the details of Redfield’s work until it has been peer reviewed and published.

So – enough of an introduction.  What is it I wanted to write about peer review?  What I want to discuss here is that the deification of a particular kind of journal peer review by the arsenic-life authors is alas not unique.  There are many who seem to have similar feelings (e.g., see this defense of the Wolfe-Simon position).  I believe this attitude towards peer review is bad for science.  Fortunately, many others agree (e.g., see this rebuttal of the defense mentioned above) and there is a growing trend to expand the concepts of what peer review is and what it means (see for example, David Dobbs great post about peer review and open science from yesterday).

Though much has been written about peer review already (e.g., see Peer review discussion at Nature as one example), I would like to add my two cents now – focusing on the exalted status some give to peer reviewed journal articles.  I have three main concerns with this attitude which can be summarized as follows

  1. Peer review is not magic
  2. Peer review is not binary
  3. Peer review is not static.

I suppose I could stop here but I should explain.

Regarding #1 “Peer review is not magic.”. 
What I mean by this is that peer review is not something that one can just ask for and “poof” it happens.  Peer review of articles (or any other type of peer review for that matter) frequently does not work as sold – work that is poor can get published and work that is sound can get rejected.  While it may pain scientists to say this (and brings up fears of FoxNews abusing findings) it is alas true.  It is not surprising however given the way articles get reviewed.

In summary this is how the process works.  People write a paper.  They then submit it to a journal. An editor or editors at the journal decide whether or not to even have it reviewed.  If they decide “no” the paper is “sent back” to the authors and then they are free to send it somewhere else.  If they decide “yes” to review it, the editors then ask a small number of “peers” to review the article (the number usually ranges from 2-3 in my field).  Peers then send in comments to the editor(s) and the editor(s) then make a “decision” and relay that decision to the authors.  They may say the paper is rejected.  Or they may say it is accepted.  Or they may say “If you address the comments of the reviewers, we would consider accepting it”.  And then the authors can make some revisions and send it back to the editors.  Then it is reviewed again (sometimes just by the editors, sometimes by “peers”).  And it may be accepted or rejected or sent back for more revisions.  And so on.

In many cases, the review by peers is insightful, detailed, useful and in the best interests of scientific progress.  But in many cases the review is flawed.  People miss mistakes.  People are busy and skim over parts of the paper.  People have grudges and hide behind anonymity.  People can be overly nice in review if the paper is from friends.  People may not understand some of the details but may not let the editors know.  Plus – the editors are not completely objective in most cases either.  Editors want “high profile” papers in many cases.  They want novelty.  They want attention.  This may lead them to ignore possible flaws in a paper in exchange for the promise that it holds.  Editors also have friends and enemies.  And so on.  In the end, the “peer review” that is being exalted by many is at best the potentially biased opinion of a couple of people.  At worst, it is a steaming pile of … Or, in other words, peer review is imperfect.  Now, I am not saying it is completely useless, as peer review of journal articles can be very helpful in many ways.  But it should be put in its rightful place.

Regarding #2: “Peer review is not binary”
The thumbs up / thumbs down style of peer review of many journal articles is a major flaw.  Sure – it would be nice if we could apply such a binary metric.  And this would make discussing science with the press and the public so much easier “No ma’am, I am sorry but that claim did not pass peer review so I cannot discuss it” “Yes sir, they proved that because their work cleared peer review.”  But in reality, papers are not “good” or “bad”.  They have good parts and bad parts and everything in between.  Peer review or articles should be viewed as a sliding scale and not a “yes” vs. “no.”

Regarding #3: “Peer review is not static”
This is perhaps the most important issue to me in peer review of scientific work.  Peer review of journal articles (as envisioned by many) is a one time event.  Once you get the thumbs up – you are through the gate and all is good forever more.  But that is just inane. Peer review should be – and in fact with most scientists is – continuous.  It should happen before, during and after the “peer review” that happens for a publication.  Peer review happens at conferences – in hallways – in lab meetings – on the phone – on skype – on twitter – at arXiv – in the shower – in classes – in letters – and so on.  Scientific findings need to be constantly evaluated – tested – reworked – critiqued – written about – discussed – blogged – tweeted – taught – made into art – presented to the public – turned inside out – and so on.
Summary:
In the end – what people should understand about peer review is that though it is not perfect, it can be done well.  And the key to doing it well is to view it as a continuous, nuanced activity and not a binary, one time event.  

UPDATE 3: some twitter comments

//platform.twitter.com/widgets.js

//platform.twitter.com/widgets.js

//platform.twitter.com/widgets.js

//platform.twitter.com/widgets.js

——–
This is from the “Tree of Life Blog”
of Jonathan Eisen, an evolutionary biologist and Open Access advocate
at the University of California, Davis. For short updates, follow me on Twitter.

——–

2010 "Arsenic found in DNA", 2012 "We never claimed arsenic was in the DNA" WTF?

Unbelievable.  Check out this news story on some new results relating to the “Arsenic Life” story.  The story discusses a paper from Rosie Redfield that has been deposited in arXiv.  Rosie has been persistent in doing tests on the strain GFAJ-1 that Wolfe-Simon had isolated.  One of their new results is that they cannot detect arsenic/arsenate in the DNA from this strain.  Amazingly, in this news story Wolfe-Simon is reported to have said that they never claimed that arsenic was getting into the DNA:

Wolfe-Simon, who says she can’t comment in detail until Redfield’s results appear in a peer-reviewed journal, wrote in an email that her original paper never actually claimed that arsenate was being incorporated in GFAJ-1’s DNA, but that others had jumped to that conclusion. “As far as we know, all the data in our paper still stand,” she wrote. “Yet, it may take some time to accurately establish where the [arsenic] ends up.”

Wow.  I recommend people go check out the original paper and see for themselves.  And also check out the press conferences and news stories.  The whole thing was about their claim that the arsenic was ending up in the DNA. 
In their abstract, for example:

Our data show evidence for arsenate in macromolecules that normally contain phosphate, most notably nucleic acids and proteins. Exchange of one of the major bio-elements may have profound evolutionary and geochemical importance.

In their conclusions:

We report the discovery of an unusual microbe, strain GFAJ-1, that exceptionally can vary the elemental composition of its basic biomolecules by substituting As for P. How As insinuates itself into the structure of biomolecules is unclear, and the mechanisms by which such molecules operate are unknown.

I personally am hoping beyond hope that Wolfe-Simon was misquoted in the new story, but I am guessing that that is unlikely.  As I have said before, I feel some sympathy towards Wolfe-Simon and I was one of the first people to call for the community to stop the personal attacks against her and to focus on the science and her claims about the science.  And I still think we need to do this.  But this does not mean we should to not criticize her claims and the almost ludicrous path she is leading some people down with her comments. The notion that they never claimed arsenic/arsenate was getting into the DNA of the strain they isolated is beyond absurd. 

Hat tip to Rosie Redfield for alerting me to this news story.

UPDATE: See these other stories on the new work

UPDATE2: Here are some additional quotes from the original paper to consider:

These measurements therefore specifically demonstrated that the purified DNA extracted from +As/–P cells contained As.

Our NanoSIMS analyses, combined with the evidence for intracellular arsenic by ICP-MS and our radiolabeled 73AsO43– experiments, indicated that intracellular AsO43– was incorporated into key biomolecules, specifically DNA

Therefore, our x-ray data support the position of AsO43– in a similar configuration to PO43– in a DNA backbone or potentially other biomolecules as well

UPDATE3: Some quotes from older news stories

From the Christian Science Monitor 12.2.2010

“So far we’ve showed that it can do it in DNA, but it looks like it can do it in a whole lot of other biomolecules” as well, says Wolfe-Simon, a NASA research fellow in residence at the USGS in Menlo Park, California. 

“It is the first time in the history of biology that there’s been anything found that can use one of the different elements in the basic structure,” says Paul Davies, the director of BEYOND: Center for Fundamental Concepts in Science at Arizona State University in Tempe, Arizona.

 From CNN 12-2-2010

“We’ve discovered an organism that can substitute one element for another,” said NASA scientist Felisa Wolfe-Simon. “Nothing should have grown. Put your plant in the dark, it doesn’t grow.” 

The bacterium not only grew but also incorporated the arsenic molecules into its DNA, in place of phosphorus, she said 

“We’ve cracked open the door to what’s possible elsewhere in the universe,” Wolfe-Simon said during a press conference Thursday.

UPDATE 4: Here is the text of one of the original press releases entitled “Get Your Biology Textbook…and an Eraser!”

One of the basic assumptions about life on Earth may be due for a revision thanks to research supported by NASA’s Astrobiology Program. Geomicrobiologist Felisa Wolfe-Simon has discovered a bacterium in California’s Mono Lake that uses arsenic instead of phosphorus in its DNA. Up until now, it was believed that all life required phosphorus as a fundamental piece of the ‘backbone’ that holds DNA together. The discovery of an organism that thrives on otherwise poisonous arsenic broadens our thinking about the possibility of life on other planets, and begs a rewrite of biology textbooks by changing our understanding of how life is formed from its most basic elemental building blocks. Astrobiology Magazine has the story. 

Wolfe-Simon’s research is supported by NASA’s Exobiology and Evolutionary Biology (Exo/Evo) Program and the NASA Astrobiology Institute. Among the goals of these programs is determining the evolution of genes, metabolic pathways, and microbial species on Earth in order to understand the potential for life on other worlds. Wolfe-Simon’s discovery represents the first time in the history of biology that an organism has been found to use a different element to build one of its most basic structures. The paper appears in today’s issue of “Science Express“ and will subsequently be published in the journal Science.

UPDATE 5: In 2010 for the press conference about the arsenic story NASA even released a video showing how arsenic could replace phosphorus in DNA.

UPDATE 6: A video of the original press conference shows Wolfe-Simon introducing the video as a model of how they think arsenic replaces phosphorus in the DNA.

UPDATE 7: In a blog post relating to the arsenic life story, Brian Krueger suggests we should in essence discount some new work by Rosie Redfield on the topic because it has not “been properly reviewed.” – see his full post here: A peril of “Open” science: Premature reporting on the death of #ArsenicLife

I tried to comment there but something did not work so I figured I would post my comments here. I think his point is completely and thoroughly wrong. What I had tried to post there I thought might be useful to share here:

I cannot disagree more with your post here. You vastly overvalue what happens in peer review. Peer review should not be considered a thumbs up / thumbs down process as you are suggesting here. And it should not be considered a one time event. It should be considered a continuous process and a sliding scale. Some things that get through the normal peer review process for papers are end up being retracted and many things that are presented prior to traditional peer review are fundamental new insights. Scientific results can be evaluated before, during and after the review that happens for a publication. Scientists do this all the time already – at conferences – in hallways – in lab meetings – on the phone – on skype – on twitter – at arXiv – in the shower – in classes – in letters – and so on. It is actually a disservice to science to annoint “peer review” as applied at some journals into something it is not.

Also see Zen Faulkes’ post in response to Brian’s: Reporting on that non peer reviewed stuff.  Hat tip to @boraz for pointing me to it.

UPDATE 8: Some links to additional stories coming out

UPDATE 9: Found a video of the whole press conference

UPDATE 10: some more links and news stories

UPDATE 11: July 7, 2012: lots of new things since March when I did the last update

UPDATE 12: Storify of Redfield’s talk at Evol2012 and related tweets


    ——–
    This is from the “Tree of Life Blog”
    of Jonathan Eisen, an evolutionary biologist and Open Access advocate
    at the University of California, Davis. For short updates, follow me on Twitter.

    ——–

    Panel from Science Online London (SOLO11): Linking with the Literature – the Arsenic Story

    Video from the panel that I skyped in to last week from my garage at 2:30 AM is now available. The detail is found at River Valley TV: Panel #1 : Linking with the Literature – the Arsenic Story | River Valley TV

    Or see it here:

    Fortunately for all I did not do skype video chat – just audio – since I was in my garage in my pajamas: Garage science: why skyping in to a meeting at 2:30 AM can be, well, interesting #SOLO11

    ——–
    This is from the “Tree of Life Blog”
    of Jonathan Eisen, an evolutionary biologist and Open Access advocate
    at the University of California, Davis. For short updates, follow me on Twitter.

    ——–

    Garage science: why skyping in to a meeting at 2:30 AM can be, well, interesting #SOLO11

    Well, that was in a way fun.  I was a part of a discussion panel for the Science Online London 2011 meeting.  However, I did not go to London … I did it from my garage.  More about this below:

    I got an email many months ago inviting me to participate in a panel discussion for a Science Online Meeting in London.  The discussion was to focus on the NASA arsenic life story and how this related to open science issues.  Now even though the one and only Kaitlin Thaney invited me, I alas said “it’s going to be hard for me to get to London at that time” and eventually we settled on the idea of me skyping in for the panel.

    And I sort of forgot about this for a bit.

    And then Joerg Heber emailed me with some comments about the panel and there was an email discussion among the panelists about details.  And in the midst of all of this Joerg pointed out that the panel was at 10:15 AM London time.  All along I had thought it was going to be in the evening in London.  10:15 AM London time is 2:15 AM California time.  And thus I started to freak out a little bit.  How was I going to pull this off?  I have two young kids.  And my house is very open in design so there is nowhere to go where people can’t hear you.  I would wake up everyone if I talked on a panel from inside my house …  hmmm

    Anyway – here are some twitter posts that will give you a feel for how things played out from a mechanical point of view:

    silentypewriter Ed Gerstner
    Getting psyched for panel discussion with @ivanoransky @rosieredfield @phylogenomics @joergheber at #solo11bit.ly/rtSwpY2 Sep

    phylogenomics Jonathan Eisen
    Perils of an open house design; am skyping in to #scio11 panel from home at 2:15 AM but have to do it from garage to avoid waking up kids
    2 Sep

    kristiholmes Kristi Holmes
    @phylogenomics I gave an online talk last year @ ORCID participants meeting in almost the same manner (3am local time) #coffeeisessential2 Sep

    phylogenomics Jonathan Eisen
    @kristiholmes the cricket in the background will be fun
    2 Sep

    phylogenomics Jonathan Eisen
    Perils of an open house design; am skyping in to #solo11 panel from home at 2:15 AM but have to do it from garage to avoid waking up kids
    2 Sep

    ryneches Russell Neches
    @phylogenomics You have a hot tub now. You should phone it in from there, like a movie villain.
    2 Sep

    sjcockell Simon Cockell
    Skyping in @phylogenomics for the panel session now…twitpic.com/6eswaa 2 Sep

    lualnu10 Marisa Alonso Nuñez
    3 present speakers and the 4th one on skype (it’s 2:30 am for him) lol #solo11 2 Sep

    LouWoodley Lou Woodley
    “I don’t think you want my video” – big star @phylogenomicsskype-ing in for panel discussion on linking with the literature #solo112  Sep

    rmounce Ross Mounce
    “I dont think you want my video” @phylogenomics LOL! (skyping in to #solo11 at 2am from the garage)
    2 Sep

    vibjpatel Vibhuti J. Patel
    Truly online conferencing at #solo11 – Skyping in a panellist from California for the discussion on Linking with the Literature #arseniclife 2 Sep

    @ryneches Russell Neches
    @phylogenomics Right. VILLAIN. Get yourself a chunky cordless phone and some dudes in sunglasses who stand around looking menacing.
    2 Sep via web

    phylogenomics Jonathan Eisen
    This is what skyping from your garage at 2:30 AM in California looks like for all you #solo11 folks post.ly/33cSm 2 Sep

    kaythaney Kaitlin Thaney
    Dedication 🙂 RT @phylogenomics: This is what skyping from your garage at 2:30 AM in California looks like #solo11 post.ly/33cSm 2 Sep

    That’s right … I skyped in from my garage.  In my pajamas.  And thus I refused the video connection.  But other than being exhausted the rest of the day, I think things worked out well.   I will post later with details of the panel and the SOLO meeting since there are some good reports out there.  And I hope audio/video will be posted some time too.

    Some links about the meeting:

    ——–
    This is from the “Tree of Life Blog”
    of Jonathan Eisen, an evolutionary biologist and Open Access advocate
    at the University of California, Davis. For short updates, follow me on Twitter.

    ——–

    Updated Again: Compilation of articles, news, blogs about the "arsenic bacteria" NASA study

    Lots of new stuff on the arsenic-bacteria front.  For those interested I am compiling some of the more useful links here:

    News stories:

    Blogs:
    • A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus
      • Felisa Wolfe-Simon, 
      • Jodi Switzer Blum, 
      • Thomas R. Kulp, 
      • Gwyneth W. Gordon, 
      • Shelley E. Hoeft,
      • Jennifer Pett-Ridge, 
      • John F. Stolz, 
      • Samuel M. Webb, 
      • Peter K. Weber, 
      • Paul C. W. Davies,
      • Ariel D. Anbar, 
      • and Ronald S. Oremland

        • Comment on “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus”

          • James B. Cotner and

            •  

          • Edward K. Hall

          Science 27 May 2011: 1201943Published online 27 May 2011

        • Comment on “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus”

          • Rosemary J. Redfield

          Science 27 May 2011: 1201482Published online 27 May 2011

        • Comment on “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus”

          • B. Schoepp-Cothenet, 
          • W. Nitschke, 
          • L. M. Barge, 
          • A. Ponce, 
          • M. J. Russell, 
          • and A. I. Tsapin

          Science 27 May 2011: 1201438Published online 27 May 2011

        • Comment on “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus”

          • István Csabai and

            •  

          • Eörs Szathmáry

          Science 27 May 2011: 1201399Published online 27 May 2011

        • Comment on “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus”

          • Stefan Oehler

          Science 27 May 2011: 1201381Published online 27 May 2011

        • Editor’s Note

          • Bruce Alberts

          Science 27 May 2011: 1208877Published online 27 May 2011

        • Comment on “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus”

          • David W. Borhani

          Science 27 May 2011: 1201255Published online 27 May 2011

        • Comment on “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus”

          • Steven A. Benner

          Science 27 May 2011: 1201304Published online 27 May 2011

        • Comment on “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus”

          • Patricia L. Foster

          Science 27 May 2011: 1201551Published online 27 May 2011

        • A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus

          • Felisa Wolfe-Simon, 
          • Jodi Switzer Blum, 
          • Thomas R. Kulp, 
          • Gwyneth W. Gordon, 
          • Shelley E. Hoeft,
          • Jennifer Pett-Ridge, 
          • John F. Stolz, 
          • Samuel M. Webb, 
          • Peter K. Weber, 
          • Paul C. W. Davies,
          • Ariel D. Anbar, 
          • and Ronald S. Oremland

          Science 2 December 2010: 1197258Published online 2 December 2010

    ——–
    This is from the “Tree of Life Blog”
    of Jonathan Eisen, an evolutionary biologist and Open Access advocate
    at the University of California, Davis. For short updates, follow me on Twitter.

    ——–

    Arsenic revisited: discussing arsenic story with a #UCDavis biology writing class next week

    Well, this could be fun. Next week I am making a guest appearance in a Writing class at UC Davis. The class focuses on writing in Biology and the instructor invited me to come in as a guest to coordinate a discussion of the arsenic paper and the coverage of it.
    When the instructor asked for reading assignments I said they should read:

    I think I probably should have suggested they read Zimmer’s excellent full write up here.  Going to suggest that now but may be too late.

    Any other pointers to good write ups of what has happened since the first week after the paper would be appreciated.

    ————-
    Some suggestions coming in from twitter:

    ——–
    This is from the “Tree of Life Blog”
    of Jonathan Eisen, an evolutionary biologist and Open Access advocate
    at the University of California, Davis. For short updates, follow me on Twitter.

    ——–