For those who missed it: "Science as an open enterprise" from Royal Society

This will be of interest to many I think: Science as an open enterprise – Report | Royal Society

It is a comprehensive report from the Royal Society with links to videos, text, previous meetings, references, EPUBs, and more relating to a report that was released a few days ago.  From the web site:

Six key areas for action are highlighted in the report:

  • Scientists need to be more open among themselves and with the public and media
  • Greater recognition needs to be given to the value of data gathering, analysis and communication
  • Common standards for sharing information are required to make it widely usable
  • Publishing data in a reusable form to support findings must be mandatory
  • More experts in managing and supporting the use of digital data are required
  • New software tools need to be developed to analyse the growing amount of data being gathered

Definitely worth a serious browsing/reading.

For more on this see …

PLOTS ( is coming to #DavisCA

DIY Science & Citizen Science & Open Science all rolled into one -PLOTS is coming to Davis, CA & UC Davis.  See  Davis – for more information.

Next up for Science in Congress: HR3433 – the Grant Reform and Transparency Act

Just got pointed to this by Mark Martin. There is a new bill making its way through congress – HR 3433 – the Grant Reform and New Transparency Act of 2011. It has a subtitle apparently of “To amend title 31, United States Code, to provide transparency and require certain standards in the award of Federal grants, and for other purposes.”
The full text of the bill and other information is available here.
I personally don’t know much about this bill but found some discussion of it here:
I have not formed an opinion of the act but thought I would share the information since it does not seem to be getting much attention but seems like it could have impact.  I note – one group that I respect deeply supports the act: the Sunlight Foundation which involves people like Ester Dyson and Lawrence Lessig.  Any opinions or insight on the bill would be welcome.

Stop deifying "peer review" of journal publications:

Peer review.  It is a critical part of scientific research and scientific progress.  Without it, science as a field might look like Fox News Stories or postings on Jenny McCarthy’s web site, where ideas people have are given gravitas regardless of how ludicrous they are.  But somehow, many in the public and press, and many many scientists alas, have deep misconceptions about peer review.

The most recent example of such misconceptions involves the arsenic life saga.  If you are not familiar with this story – here is a summary (for some fine scale details on the early parts of the story see Carl Zimmer’s post here).

In November 2010 NASA announced that in a few days they would hold a press conference discussing a major finding about life in the universe.  On December 2, 2010, they held their press conference and discussed a paper that was in press in Science from multiple NASA funded authors including Felisa Wolfe-Simon.  The paper was of interest because it claimed to have shown that a bacterium was able to replace phosphate in its macromolecules, including its DNA, with arsenic.  The press conference made claims that were very grandiose, like that textbooks would have to be rewritten, and the study of life on Earth and elsewhere would have to be completely rethought.

After a few days of mostly very glowing press reports, a few critiques began to emerge including in particular one from Rosie Redfield, a microbiologist at the University of British Columbia.  The critiques then snowballed and snowballed and the general consensus of comments appeared to be that the paper had fundamental flaws.  Some of the critiques got way too personal in my opinion and I begged everyone to focus on the science not personal critiques.  This seemed to work a little bit and we could focus on the science, which still seemed to be dubious.  And many, including myself, expressed the opinion that the claims made by the authors in the paper and by the authors and NASA in the press conference and in comments to the press, were misleading at best.

Now critiques about new findings are not unusual.  We will get back to that in a minute.  But what was astonishing to me and many others, was how NASA and the authors responded.  They said things like:

… we hope to see this work published in a peer-reviewed journal, as this is how science best proceeds.


It is one thing for scientists to “argue” collegially in the public media about diverse details of established notions, their own opinions, policy matters related to health/environment/science. 

But when the scientists involved in a research finding published in scientific journal use the media to debate the questions or comments of others, they have crossed a sacred boundary [via Carl Zimmer]

and the kicker for me was a letter Zimmer posted

Mr. Zimmer, 

I am aware that Dr. Ronald Oremland has replied to your inquiry. I am in full and complete agreement with Dr. Oremland’s position (and the content of his statements) and suggest that you honor the way scientific work must be conducted. 

Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated. You can see many examples in the journals Science and Nature, the former being where our paper was published. This is a common practice not new to the scientific community. The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner. 


This was amazing since, well, they were the ones who held the overhyped press conference.  And then I (and others) found it appalling that they in essence would not response to critiques because they were not “peer reviewed.” I told Zimmer

Whether they were right or not in their claims, they are now hypocritical if they say that the only response should be in the scientific literature.

Zimmer had a strong defense of scientists “discussing” the paper:

Of course, as I and others have reported, the authors of the new paper claim that all this is entirely inappropriate. They say this conversation should all be limited to peer-reviewed journals. I don’t agree. These were all on-the-record comments from experts who read the paper, which I solicited for a news article. So they’re legit in every sense of the word. Who knows–they might even help inform peer-reviewed science that comes out later on.

(I note – yes I am quoting a lot from Zimmer’s articles on the matter and there are dozens if not hundreds of others – apologies to those out there who I am not referencing – will try to dig in and add other references later if possible).

And so the saga continued.  Rosie Redfield began to do experiments to test some of the work reported in the paper.  Many critiques of the original paper were published.  The actual paper finally came out.  And many went about their daily lives (I keep thinking of the Lord of the Rings whisper “History became legend. Legend became myth. And for two and a half thousand years, the ring passed out of all knowledge.”  Alas, the arsenic story did not go away.

And now skipping over about a year.  The arsenic story came back into our consciousness thanks to the continued work of Rosie Redfield.  And amazingly and sadly, Wolfe-Simon’s response to Rosie’s work included a claim that they never said that arsenic was incorporate into the bacterium’s DNA.  (I have posted a detailed refutation of this new “not in DNA” comment here).

But that is not what I am writing about here.  What is also sad to me are the continued statements by the paper’s authors that they will not discuss any critiques or work of others unless they are published in a peer reviewed article.

For example, see Elizabeth Pannisi’s article in Science:

But Wolfe-Simon and her colleagues say the work on arsenic-based life is just beginning. They told ScienceInsider that they will not comment on the details of Redfield’s work until it has been peer reviewed and published.

So – enough of an introduction.  What is it I wanted to write about peer review?  What I want to discuss here is that the deification of a particular kind of journal peer review by the arsenic-life authors is alas not unique.  There are many who seem to have similar feelings (e.g., see this defense of the Wolfe-Simon position).  I believe this attitude towards peer review is bad for science.  Fortunately, many others agree (e.g., see this rebuttal of the defense mentioned above) and there is a growing trend to expand the concepts of what peer review is and what it means (see for example, David Dobbs great post about peer review and open science from yesterday).

Though much has been written about peer review already (e.g., see Peer review discussion at Nature as one example), I would like to add my two cents now – focusing on the exalted status some give to peer reviewed journal articles.  I have three main concerns with this attitude which can be summarized as follows

  1. Peer review is not magic
  2. Peer review is not binary
  3. Peer review is not static.

I suppose I could stop here but I should explain.

Regarding #1 “Peer review is not magic.”. 
What I mean by this is that peer review is not something that one can just ask for and “poof” it happens.  Peer review of articles (or any other type of peer review for that matter) frequently does not work as sold – work that is poor can get published and work that is sound can get rejected.  While it may pain scientists to say this (and brings up fears of FoxNews abusing findings) it is alas true.  It is not surprising however given the way articles get reviewed.

In summary this is how the process works.  People write a paper.  They then submit it to a journal. An editor or editors at the journal decide whether or not to even have it reviewed.  If they decide “no” the paper is “sent back” to the authors and then they are free to send it somewhere else.  If they decide “yes” to review it, the editors then ask a small number of “peers” to review the article (the number usually ranges from 2-3 in my field).  Peers then send in comments to the editor(s) and the editor(s) then make a “decision” and relay that decision to the authors.  They may say the paper is rejected.  Or they may say it is accepted.  Or they may say “If you address the comments of the reviewers, we would consider accepting it”.  And then the authors can make some revisions and send it back to the editors.  Then it is reviewed again (sometimes just by the editors, sometimes by “peers”).  And it may be accepted or rejected or sent back for more revisions.  And so on.

In many cases, the review by peers is insightful, detailed, useful and in the best interests of scientific progress.  But in many cases the review is flawed.  People miss mistakes.  People are busy and skim over parts of the paper.  People have grudges and hide behind anonymity.  People can be overly nice in review if the paper is from friends.  People may not understand some of the details but may not let the editors know.  Plus – the editors are not completely objective in most cases either.  Editors want “high profile” papers in many cases.  They want novelty.  They want attention.  This may lead them to ignore possible flaws in a paper in exchange for the promise that it holds.  Editors also have friends and enemies.  And so on.  In the end, the “peer review” that is being exalted by many is at best the potentially biased opinion of a couple of people.  At worst, it is a steaming pile of … Or, in other words, peer review is imperfect.  Now, I am not saying it is completely useless, as peer review of journal articles can be very helpful in many ways.  But it should be put in its rightful place.

Regarding #2: “Peer review is not binary”
The thumbs up / thumbs down style of peer review of many journal articles is a major flaw.  Sure – it would be nice if we could apply such a binary metric.  And this would make discussing science with the press and the public so much easier “No ma’am, I am sorry but that claim did not pass peer review so I cannot discuss it” “Yes sir, they proved that because their work cleared peer review.”  But in reality, papers are not “good” or “bad”.  They have good parts and bad parts and everything in between.  Peer review or articles should be viewed as a sliding scale and not a “yes” vs. “no.”

Regarding #3: “Peer review is not static”
This is perhaps the most important issue to me in peer review of scientific work.  Peer review of journal articles (as envisioned by many) is a one time event.  Once you get the thumbs up – you are through the gate and all is good forever more.  But that is just inane. Peer review should be – and in fact with most scientists is – continuous.  It should happen before, during and after the “peer review” that happens for a publication.  Peer review happens at conferences – in hallways – in lab meetings – on the phone – on skype – on twitter – at arXiv – in the shower – in classes – in letters – and so on.  Scientific findings need to be constantly evaluated – tested – reworked – critiqued – written about – discussed – blogged – tweeted – taught – made into art – presented to the public – turned inside out – and so on.
In the end – what people should understand about peer review is that though it is not perfect, it can be done well.  And the key to doing it well is to view it as a continuous, nuanced activity and not a binary, one time event.  

UPDATE 3: some twitter comments





Interesting new metagenomics paper w/ one big big big caveat – critical software not available "

Very very strange.  There is an interesting new metagenomics paper that has come out in Science this week.  It is titled “Untangling Genomes from Metagenomes: Revealing an Uncultured Class of Marine Euryarchaeota” and it is from the Armbrust lab at U. Washington.

One of the main points of this paper is that the lab has developed software that apparently can help assemble the complete genomes of organisms that are present in low abundance in a metagenomic sample.  At some point I will comment on the science in the paper, (which seems very interesting) though as the paper in non Open Access I feel uncomfortable doing so since many of the readers of this blog will not be able to read it.

But something else relating to this paper is worth noting and it is disturbing to me.  In a Nature News story on the paper by Virginia Gewin there is some detail about the computational method used in the paper:

“He developed a computational method to break the stitched metagenome into chunks that could be separated into different types of organisms. He was then able to assemble the complete genome of Euryarchaeota, even though it was rare within the sample. He plans to release the software over the next six months.”

What?  It is imperative that software that is so critical to a publication be released in association with the paper.  It is really unacceptable for the authors to say “we developed a novel computational method” and then to say “we will make it available in six months”.  I am hoping the authors change their mind on this but I find it disturbing that Science would allow publication of a paper highlighting a new method and then not have the method be available.  If the methods and results in a paper are not usable how can one test/reproduce the work?

Dear #AAAS, I am NOT embargoing my own talk & I plan to record it and post afterwards #embargowatch

Just got another email from AAAS regarding their big meeting in February in Vancouver where I am scheduled to talk:

This request for materials is from the AAAS media relations team and is separate from any you may receive from your symposium organizer or the AAAS Annual Meeting office.
Dear AAAS Annual Meeting Participant:

Thanks to all of you who uploaded materials to the AAAS Virtual Newsroom by Jan. 16. For those of you who have not submitted materials or want to submit additional materials, you may do so right up through the meeting. The materials will be available online to reporters, although we can no longer guarantee that we’ll be able to copy new
submissions at our expense for placement in the on-site library of speaker materials. We will try to include materials received in the next several days in our copy order, however.

You also can make printed copies (10-15 copies) yourself and ship them to Vancouver so that we can place them in the on-site papers library for reporters. Ideally, press materials should be on-site prior to your presentation. Please see below for appended mailing instructions.

Speakers and organizers can submit materials by going to:

Your individual username and password for the site:

Please provide the following:

— A one-paragraph biographical sketch (not a C.V.)

— A short lay-language summary of your talk, beyond the abstract.

— The text of your talk, if available, or a related (ideally recent) technical paper, either as a Word file or a PDF. PowerPoint presentations are acceptable, but a full text will better serve reporters’ needs.

— Any additional supporting materials, including multimedia files such as JPEG or TIFF photos in high resolution (300 dpi) and/or digitized video clips.

IMPORTANT: Please note that all AAAS meeting presentations are strictly embargoed and your speaker materials should not be released publicly until the time of your presentation.

If you upload your materials by 16 January, we will copy them at our expense for placement in the on-site library of speaker materials, available only to newsroom registrants.

Please notify your institution’s press office of your AAAS Annual Meeting presentation as soon as possible. Your press office can help you submit speaker materials to us and can begin to generate media interest.

The thing is – I did not agree to “Embargo” my talk and as I wrote about before, I do not even know what that means.  I figured, in the interest of being “open” about my feelings about this, I should write to AAAS to let them know I was not going to embargo my own talk, and I plan to record my talk and post it afterwards:

To whom it may concern

I am scheduled to speak at the AAAS meeting and I am writing this in regard to the email attached below.  I do not support the notion of an Embargo for my talk and I am unwilling to participate in the embargo. I plan to post information about my talk to the web and to my blog and am writing to specifically let you know I fundamentally do not support the embargo nor did I agree to it when I agreed to give a talk at AAAS.

I also plan to record my own talk and to post the recording and the slides to various websites.  I am not sure if AAAS has a policy about that but wanted to let you know of my plans in the interest of not having any surprises.


Jonathan Eisen

Will report back if I get a reply … and maybe I can get Ivan Oransky to help make sense out of what a talk embargo means.

Request for input – are there any rules regarding posting text of one’s own NSF (or other) grant proposals?

In response to a series of posts from Karen James (who is a biologist now in Maine and is director of the HMS Beagle Project) on Twitter, I am posting here to ask for input from the crowd.  On Twitter, Karen has been discussing her putting together an NSF proposal and was then celebrating a few days ago when it was done.

I have posted some of the twitter conversation below.  But to get directly to the point the question I have for everyone here is – are there any rules at the National Science Foundation that would prevent one from sharing with others a grant proposal that one has submitted?  Are there any rules against this at any agency?  I think there are none but apparently some are telling Karen otherwise.

Any information on this would be useful. Some of the twitter conversation is below:

So, @phylogenomics and others, with whom is it appropriate to share a submitted NSF proposal? Anyone? No one? Something in between?
1/12/12 6:15 PM

@kejames what do you mean by “appropriate”
1/12/12 6:15 PM

@kejames If I were you, I’d share the whole thing in public and make the reviews public as well. But I’m a minority view.
1/12/12 6:16 PM

@johnhawks Thanks, I’ve thought of that, actually. It is a federal agency after all. I’d need to redact confidential budget info, though.
1/12/12 6:18 PM

@kejames Yes, and possibly key personnel. My attitude is the success rate is so low, it can’t hurt and might draw visibility pre-review.
1/12/12 6:19 PM

@phylogenomics Best practice. My instinct is to share it with colleagues, collaborators and associates I think might be interested in it.
1/12/12 6:16 PM

@kejames what is the potential reason to not share?
1/12/12 6:17 PM

@phylogenomics That’s what I’m asking. Is there any rule or custom that prohibits sharing it far and wide?
1/12/12 6:19 PM

@kejames none that I know of – only reason not to is if you are worried about people “stealing” your ideas
1/12/12 6:19 PM

@phylogenomics Not worried about that in the slightest. If anything sharing it widely establishes it as “my” idea. Thanks.
1/12/12 6:20 PM

@johnhawks Or I could ask the key personnel if they’re okay w/ it. I think it would be nice to include them if they want to be included.
1/12/12 6:21 PM

@kejames I think Rosie Redfield posts hers on her blog
1/12/12 6:25 PM

@phylogenomics Thanks for that. I notice she just posts the project description itself, none of the other “stuff”and doesn’t list names.
1/12/12 6:28 PM

@kejames NSF proposals are your choice who to share with. Probably not best to post publicly, though.
1/12/12 6:25 PM

@DoctorZen Why not? As @phylogenomics notes, @RosieRedfield posts her grant proposals on her lab’s website:
1/12/12 6:29 PM

Anyone else besides @phylogenomics @doctorzen @johnhawks want to weigh in on how broadly I should share my just-submitted NSF proposal? 1/2
1/12/12 6:31 PM

@rdmpage @RosieRedfield @DoctorZen @johnhawks @phylogenomics @kzelnio So I asked the collaborators on the proposal. One replied… 1/2
1/13/12 4:50 AM

@rdmpage @RosieRedfield @DoctorZen @johnhawks @phylogenomics @kzelnio 2/2…”Sharing not wise! Could disqualify proposal.”
1/13/12 4:51 AM

.@kejames @rosieredfield @doctorzen @johnhawks @phylogenomics @kzelnio In other words fear of what grant agency will do trumps being open 😦
1/13/12 5:10 AM

@rdmpage I support being open; not sure every step always needs to be public. @kejames @rosieredfield @johnhawks @phylogenomics @kzelnio
1/13/12 5:18 AM

.@DoctorZen @kejames @rosieredfield @johnhawks @phylogenomics @kzelnio I agree, it’s not that it HAS to open, but that it COULD be
1/13/12 5:27 AM

@kejames @rdmpage @RosieRedfield @DoctorZen @johnhawks @kzelnio WTF? As far as I know there are NO NSF issues w/ sharing a proposal
1/13/12 7:00 AM

@kejames @rdmpage @RosieRedfield @DoctorZen @johnhawks @kzelnio Yes, need to discuss w/ collabs & get permission but not angst any rules
1/13/12 7:02 AM

@rdmpage @DoctorZen @kejames @rosieredfield @johnhawks @kzelnio agree w/ Rod – issue was whether it could be posted, not if it had to be
1/13/12 7:04 AM

@phylogenomics I’m following up w/ him to find out what he meant He’s a seasoned NSF grantee and reviewer. Have also contacted NSF directly.
1/13/12 7:08 AM

Draft post cleanup #3: The Open Knowledge Foundation

Yet another post in my “draft blog post cleanup” series.  Here is #3 from just a few weeks ago:

Interesting article in PLoS Biology:  PLoS Biology: The Open Knowledge Foundation: Open Data Means Better Science.  It discusses many issues in open science especially as they relate to open data.

Some links from this paper are worth checking out

This article reminds me that I keep meaning to push for the development of a “Datawatch” system much like the “RetractionWatch” systems of Ivan Oransky. I have discussed this with Ivan but we have not yet gotten around to doing it … 

Garage science: why skyping in to a meeting at 2:30 AM can be, well, interesting #SOLO11

Well, that was in a way fun.  I was a part of a discussion panel for the Science Online London 2011 meeting.  However, I did not go to London … I did it from my garage.  More about this below:

I got an email many months ago inviting me to participate in a panel discussion for a Science Online Meeting in London.  The discussion was to focus on the NASA arsenic life story and how this related to open science issues.  Now even though the one and only Kaitlin Thaney invited me, I alas said “it’s going to be hard for me to get to London at that time” and eventually we settled on the idea of me skyping in for the panel.

And I sort of forgot about this for a bit.

And then Joerg Heber emailed me with some comments about the panel and there was an email discussion among the panelists about details.  And in the midst of all of this Joerg pointed out that the panel was at 10:15 AM London time.  All along I had thought it was going to be in the evening in London.  10:15 AM London time is 2:15 AM California time.  And thus I started to freak out a little bit.  How was I going to pull this off?  I have two young kids.  And my house is very open in design so there is nowhere to go where people can’t hear you.  I would wake up everyone if I talked on a panel from inside my house …  hmmm

Anyway – here are some twitter posts that will give you a feel for how things played out from a mechanical point of view:

silentypewriter Ed Gerstner
Getting psyched for panel discussion with @ivanoransky @rosieredfield @phylogenomics @joergheber at Sep

phylogenomics Jonathan Eisen
Perils of an open house design; am skyping in to #scio11 panel from home at 2:15 AM but have to do it from garage to avoid waking up kids
2 Sep

kristiholmes Kristi Holmes
@phylogenomics I gave an online talk last year @ ORCID participants meeting in almost the same manner (3am local time) #coffeeisessential2 Sep

phylogenomics Jonathan Eisen
@kristiholmes the cricket in the background will be fun
2 Sep

phylogenomics Jonathan Eisen
Perils of an open house design; am skyping in to #solo11 panel from home at 2:15 AM but have to do it from garage to avoid waking up kids
2 Sep

ryneches Russell Neches
@phylogenomics You have a hot tub now. You should phone it in from there, like a movie villain.
2 Sep

sjcockell Simon Cockell
Skyping in @phylogenomics for the panel session now… 2 Sep

lualnu10 Marisa Alonso Nuñez
3 present speakers and the 4th one on skype (it’s 2:30 am for him) lol #solo11 2 Sep

LouWoodley Lou Woodley
“I don’t think you want my video” – big star @phylogenomicsskype-ing in for panel discussion on linking with the literature #solo112  Sep

rmounce Ross Mounce
“I dont think you want my video” @phylogenomics LOL! (skyping in to #solo11 at 2am from the garage)
2 Sep

vibjpatel Vibhuti J. Patel
Truly online conferencing at #solo11 – Skyping in a panellist from California for the discussion on Linking with the Literature #arseniclife 2 Sep

@ryneches Russell Neches
@phylogenomics Right. VILLAIN. Get yourself a chunky cordless phone and some dudes in sunglasses who stand around looking menacing.
2 Sep via web

phylogenomics Jonathan Eisen
This is what skyping from your garage at 2:30 AM in California looks like for all you #solo11 folks 2 Sep

kaythaney Kaitlin Thaney
Dedication 🙂 RT @phylogenomics: This is what skyping from your garage at 2:30 AM in California looks like #solo11 2 Sep

That’s right … I skyped in from my garage.  In my pajamas.  And thus I refused the video connection.  But other than being exhausted the rest of the day, I think things worked out well.   I will post later with details of the panel and the SOLO meeting since there are some good reports out there.  And I hope audio/video will be posted some time too.

Some links about the meeting:

My science communication hero/heroine of the month – Dr. Kiki @drkiki

Been working on revising my lab’s web site and was looking for some videos of talks I have given online to post there.  And I discovered/rediscovered this video of an interview I did for Dr. Kiki’s Science Hour.  Here it is:


Now I know – this is over a year old. But I just watched the full video. Not so bad I think.

As many of you know, I like to talk.  And talk.  And talk.  But I would like to say that as an interviewer, Dr. Kiki is pretty frigging awesome.  Don’t know how she does it.  But I am going to post this video on the new lab page and point people to it if they want to know what my lab does and what I am interested in.

But enough about me.  I want to thank Dr. Kiki for this great interview by saying a little bit about her.  Or, well, her work in science communication.

As some of you may know, I listen to podcasts of TWIS – This Week in Science frequently on my bike rides to work.  And I really recommend anyone/everyone out there give it a whirl.  It is sort of like Science Friday but it is a bit edgier, a bit funnier, a bit goofier, and a bit sciencier (is that a word?)  Dr. Kiki and Justin on it are great and it is so good that I frequently sit outside my building listening to the end of a show if I take the short ride to work which is less than an hour.  So if you like Science – you really should check out the TWIS web site and find some way to listen such as what I do by subscribing to their podcasts at iTunes.

And I guess now I will be checking out “Dr. Kiki’s Science Hour” more after rewatching this video.  There are many many more shows at  I have not checked out as many as TWIS shows but the ones I have watched are great.

And if you want to follow her more directly check out her Blog: The Bird’s Brain, or her twitter feed  (@drkiki)  or her  Google+ feed.

Very proud that she is a UC Davis alum … and just want to say thanks to her for giving me a video I can share with others that says more about me and my lab than almost anything I have written.