Case Study – what to do when there are disagreements about whether a paper has problems

In light of the ongoing PLOSOne #Creationgate controversy (e.g. see this write up here), I thought I would share a story I have been working on about a case where there are disagreements about whether a paper has problems or not.

So I got this email the other day. It was from an author of a paper who I know who was upset about a paper that was published a while back for which I was the editor. This person wrote to me and the authors of the new paper somewhat angrily critiquing them for some aspects of their paper that related to this person’s work. The authors responded to the critique and, well, did not agree with the points of the letter writer. The letter writer wrote again to all of us and again somewhat angrily critiqued the authors.

Then the letter writer wrote just to me asking what I thought should be done – detailing further what they viewed as mistakes of the paper.  The letter writer was quite clear, clearly upset, and had some good points. And the letter writer wanted advice about what should be done here.  I thought about this for some time and wrote and rewrote an email to the letter writer.  The challenge with this case was that this really seemed to be more of a disagreement than a case where an Editor could say “This is right and this is wrong.”  So this is what I wrote

Letter Writer

I appreciate your comments and your intent here.  I think the best course of action is for you to publish more public, formal or informal, comments about the paper.  I am not sure I would support any type of attempt to require the authors to officially revise their paper.  The paper was reviewed by multiple reviewers and the published version is the final outcome of the review process.  I believe the process was fair, rigorous and thorough.  That does not mean of course that it was perfect (note – I am not making any statement here about whether I think your comments and concerns are valid or not).  But as far as I can tell, even if your claims and comments were 100% valid and correct, I probably would still not recommend undertaking any action that would require modifications of the paper by the authors.  I therefore would recommend that you pursue other options including 

(1) submitting comments about this on the journal web site 
(2) submitting comments at Pubmed Commons or other such sites 
(3) submitting a formal response via the journal 

If you wish to pursue further the possibility of requesting a modification of the paper by the authors, I am happy to forward this on to the higher ups at the journal and they can tell you how to do this. But as I note, I am not sure this would be the right thing to do here.

Jonathan Eisen

I don’t know if I did the right thing here but I just felt that this was a case where people could just disagree about what was correct.  Any suggestions for how to handle such cases or other examples would be welcome. 

Has your scientific research been wasted?

I had a good Thanksgiving weekend this year – spending time with family and friends. But as I go back to work this week I have now gotten somewhat depressed over something I did Sunday night. I decided to remove myself from the UC Davis internet proxy to see how many of my past papers that I have published I can obtain without the UC subscriptions. So I went to pubmed, and typed in my name (Eisen JA) and got most of my papers, which are listed at the bottom of this blog (some do not come up due to publication off the pubmed grid or due to co-authors screwing up my initials). (NOTE  – LISTING DELETED 4/09 BECAUSE THE FORMATTING IS ALL MESSED UP)

And then I went to see how many of my papers were freely available and how many were not. What I was most interested in was – what is the deal with papers I wrote before becoming an Open Access convert? For many it is easy to figure out if they are freely available – Pubmed has a link saying “Free in PMC” which refers to Pubmed Central. For others, it was a little trickier.

The results were both good and bad and a summary is below. A few things struck me. First, a lot of my life’s work is not readily available without paying other for it. In the day and age of the internet, this means that these papers will simply be read less and less as time goes by. And that makes me very sad. If I had chosen to publish those papers in other journals, anyone in the world could get them at any time. Thankfully I did publish many papers in journals like PNAS, and ASM journals, and NAR – journals that have now decided to release them to Pubmed Central. And also thankfully (but less so) I published some papers in journals that have at least made them freely available on their web sites.

Most surprisingly to me is that a reasonable number of my papers in Nature are freely available on the Nature web site as part of their Genomics Gateway program. Nature deserves serious kudos for doing this and they stand out compared to Elsevier journals (which do not seem to ever do this) and even Science. This is disappointing as Science is published by a scientific society but apparently does not seem to care much about access to publications. Nature, a commercial publisher, is in my opinion doing more for scientific openness than Science. Now, Nature has a long way to go, but I am SO glad I listened to their editors like Chris Gunter and Tanguy Chouard who made a big deal about the Genome papers being free. I did not think it was that big a deal, but in retrospect they were ahead of me in thinking about availability. Plus Nature clearly makes more of an effort to provide free online material than they have to – and certainly make more available than Science.

So in the end – I am sad about my partially wasted past. But I am pleasantly surprised that at least some papers I thought would be more restricted are actually free (although only on the Publishers site for now – Hopefully these journals will submit them to PMC at some point). I guess – you win some and you lose some and some are somewhere in between.

Summary of openness — other scientists should do this exercise

In Pubmed Central and Open Access

Available free on publisher’s sites (notideal but better than nothing)

Must buy paper

Not available anywhere

Good Open Access Biology Resources

Boring blog overall, but I wanted to put a collection of links here for information about Open Access, especially as it regards to biomedical literature. I will add more links to this over time, and welcome suggestions.

Royal Society just digs a deeper hole

The Royal Society has announced that they are making their full archive, including papers going back hundreds of years, available online for the first time. I read this line and thought – “Finally, the Royal Society is moving towards Open Access”. After all, the US National Academy of Sciences provides full and free access to all articles 6 months after publication.

Then I read the next sentence, which says that the Royal Society wil provide this free access to their archive until December.:

And until December the archive is freely available to anyone on the internet to explore. ….

After December 2006 subscribers to our subscription packages (S, A and B) will enjoy privileged online access to the archives. Private researchers will also be able to access individual articles for a small fee per download.

The Royal Society appears to simly want to hold on to every little last shred of money they can get for things published originally hundreds of years ago. They could make a great contribution to the world by opening up their archive completely. But clearly, the Royal Society is not about making contributions to humanity. What they appear to be about is a scientific oligarchy that exists mostly to promote themselves and their freinds. I would like to point out again that of 1316 fellows, 62 are women.

So this group of scientists appears to be trying to continue the bad traditions started hundreds of years ago, like excluding women from science. I looked for but could not find information on minorities but can only assume that their record in this area is even worse, as they do not discuss it on their web site.

Perhaps some day the UK public will wisen up and stop giving money to this collection of Neanderthal wannabes.

The hypocrisy of most projects with "Open" data release

There has been a growing trend in biological research, for scientists to release their data in some way or another prior to publication. This data release is meant to promote the advancement of science, and it frequently does. This is perhaps best seen with genome sequencing projects, such as the public version of the “Human Genome Project.” In many if not most cases, centers that do the bulk of the sequencing work release the sequence data for searching by others, even before publishing papers on their own data. In most cases, restrictions are placed on how the data can be used, but the data is still released for others to look at.

This is of course in contrast to how much of science works, with researchers keeping their data to themselves until they are ready to publish something. The genome centers who have made their data available prior to publication deserve some credit for this openness. Especially since the data release in general by genome centers has been so far and beyond what biology researchers do. In fact, many of these centers go out of their way to promote getting such credit (they even got Clinton and Blair to play along) The best example of this was the public human genome project, which made multiple claims about how great they were for humanity for releasing the data “within 24 hours of gathering it.” This data release policy was captured in something that became known as the Bermuda Principles, due to a meeting that took place in Bermuda (see a nice summary of this by John Sulston here).

What is appalling to me, however, is that these same centers that try to take credit for their openness, then turn around and usually publish their papers in non Open Access journals (for those who do not know, this means that then one has to pay money, frequently enormous sums of money, just to read the paper). I do not understand this. A paper about an analysis someone did on a data set may in fact be more valuable to the community than the data itself. If the genome centers like TIGR, JGI, Sanger, Whitehead, etc. really wanted to be on the side of openness, they should stop publishing their papers in non Open Access journals. Unfortunately these places publish very few of their papers in such journals.

For example, the Joint Genome Institute (JGI) which I am now affiliated with, is continually showing two faces on this issue. On the one hand, the issue press release after press release regarding their release of data on various genome projects (e.g., here). That is fine, although a little over the top sometimes. But then they almost never publish any of their work in Open Access journals (e.g., see their latest press release on a paper published about a genome in Science, a non Open Access journal). Any taxpayers out there should be disappointed with this as the genome centers get TONS of money to carry out this work for the public benefit. And then for the papers on the work to be hidden behind huge subscription fees is a waste of your money.

This is particuarly surprising coming from JGI since JGI is run directly by the Department of Energy (unlike most other centers which are either private or part of a university). Thus apparently DOE does not want to follow even the recommendations of congress and the senate regarding Open Access to publications. Nor does DOE apparently want to do the right thing by requiring their institutes for publish in Open Access journals. Too bad. Taxpayers hopefully will begin to get more and more upset about the waste of their money as these centers take enormous amounts of the federal science budget and convert it into documents that only a few can read.

The Blogger World Favors Open Access Publications

Well, even though the traditional press did not pick up the story about the Tetrahymena genome paper, it seems that lots of blogs and online news sources picked it up.

Here are some:

Maybe the press release from TIGR did not excite the “real” press too much, I do not know. But nevertheless, it is good to see people discussing the article and even better to see that the article is currently the #1 viewed article for the week at PLoS Biology. I asumme that most of this comes from slashdot running an item about the article but I am not 100% sure.

I think the blogger world seems to run stories about Open Access publications much more than
about non Open Access publications since they can read them freely. It would seem that the blogger world is helping to promote Open Access papers and may explain why in the recent past I have gotten much more response to Open Access papers than even to papers in Nature or Science.

It is so important for scientific research to reach all people, not just scientists who can afford subscriptions to journals. Thus a partnership between bloggers and open access publications seems perfect for the new way of doing science.

The Disgrace of the Royal Society

I am astonished at the behavior of the Royal Society regardling publication. As dozens of funding agencies and societies and individuals move towards Open Access for publications, the Royal Society crawls back into the medeivel hole from which it originated.


For example, in a recent article from RSC:

But the Royal Society of Chemistry’s director of publishing, Peter Gregory, disagrees. ‘We have absolutely no interest shown from our editorial board members, or our authors, for open access publishing,’ he said.

Gregory believes that the open access author-pays model is ‘ethically flawed’, because it raises the risk that substandard science could be widely circulated without being subjected to more rigorous peer review. This could be particularly problematic in chemistry, where rapid, open access publication could be used to establish priority ahead of more time-consuming patent applications from rival groups, he added.

What this basically means is that the Royal Society wants to continue to make money publishing the results of scientific research that is largely funded by the government and the public. And that they are willing to have people suffer (e.g., die unnecessarily because their doctors do not have a subscription to the Royal Societies journals) rather than use their supposedly brilliant minds to come up with a way to make money and simultaneously make the research freely available. The NIH, Wellcome Trust, and dozens of other groups are pushing for Open Access. Yet the Royal Society is sticking to their old boys club ways (to see how old boys clubbish they are go to here).

If we actually go to the details of the Gregory quote above, I have a hard time knowing where to begin with the flawed logic here. For example, the idea that substandard science does not get published in non Open Access journals is just absurd. Consider the latest example of the Korean Cloning scam. Those articles were published in top non open access journals. Same thing with just about every other case of bad science or scientific fraud in the last twenty years. The claim by Gregory is simply unfounded. First, Open Access journals do not say there should be no peer review and they tend to be peer reviewed even more carefully than non-open access journals. Just try publishing a paper in PLoS Biology, which I have found to be more stringent than Science. Why is this? Becuase scientists are more willing to commit time to reviewing for such journals because their work benefits humanity rather than some publisher like Gregory.

Another reason Gregory’s claim is unfounded is evidenced by the physics community. They put preprints out for the world to see, which allows for global peer review, rather than peer review by a select list of people. The idea that peer review as it is in current non open access journals is perfect is completely ridiculous. Sometimes you get objective reviewers, but other times you get people that, even if they wished to be objective, would probably have a hard time doing so. This is unavoidable in any peer review system. The more open the publication system and the peer review system is, the more likely it is to avoid outrageous variation in quality.

The Royal Society should be ashamed. They are preventing the distribution of scientific findings and trying to maintain a publishing system that limits the speed of scientific advances and enriches the publishers at the expense of governments and the public.

So I suggest that anyone who knows someone harmed by a doctor who did not know what they were doing, or anyone who wishes for scientific advancement to proceed at a rapid pace, to consider writing to your favorite member of the Royal Society and asking how they feel about this.

To contact the Royal Society directly go here.

I have been unable to come up with email lists of society members but if anyone can find one I will post it.

Open Access Rant: How Does Your Doctor Learn About the Newest Medical Findings??

Everybody would like to find a doctor who is knowledgable about the latest developments in medicine. Whether these developments relate to new treatments, or new methods of diagnosis, or treatments that are dangerous or do not work, we want our doctors to know this information. How do doctors find out about these things?

Well, there are many sources of this information, but one we hear a lot about these days is a little disconcerting. It turns out that a lot of doctors get the latest information from drug company reps who stop by the office and leave imformation pamphlets or who talk up their companies latest products. This could be OK, except for the fact that many of the drug company reps either purposefully provide misleading information, or in fact do not actually know what is good or bad information.

One reason this is such a big problem is that, like everyone else these days, doctors are really busy and overwhelmed. So they sometimes do not have any time to read the actual medical studies that might be relevant to what the drug company reps are saying. But that is a bit lame of an excuse, since it is their job to know these things. Thus I really think they should read more of the medical literature and not just drug company propaganda.

But herein is one of the biggest problems in modern medicine. Even if you have a really hard working doctor who is willing to read the latest papers, they may not be able to. This is because even though most of the medical studies were paid for by the government in some way, they are not freely available for the doctors to read, because they are published in journals that charge exceptionally high prices for subscriptions. Doctors in large institutions probably have good access to this information. But doctors in small groups may not. Imagine if congress passed laws but lawyers were not allowed to read them without paying a fee to someone. The system for medical literature is really absurd.

I got thinking about this when re-reading Lance Armstrong’s autobiography “It’s Not about the Bike.” In the book, Armstrong describes how when he had testicular cancer he had a friend who was a doctor bring him the latest studies on this type of cancer and he read all of them. Well, this only was possible because his friend must have had access to all the publications through a university or very large medical group. Wouldn’t it have been better if Armstrong could have just gotten the studies himself, given that most were paid for by the US Government in the first place? Well, if people doing medical research published their finding in Open Access journals, then anyone could read the articles, from doctors, to patients, to family members, to journalists. We would all benefit if this was done.

Powerpoint slides from Nature and others do not give full credit to the sources of images

Well, I browsed around the Nature web site and did some searches for terms like “Reprinted with permission” and then looked at how they handled Figures that were reprinted from other places. I found some additional examples where the Figure image did not seem to do a complete job of crediting the source of the material. But without a doubt the most disturbing thing I found is that you can download powerpoint slides of the figures and all the ones I looked at only had “Copyright Nature” or something like that and no information crediting the actual source of the material. This is basically because they do not include the Figure legends on the ppt slides. I am not sure if technically they are allowed to do this in some cases (my gut feeling is there is something wrong with what they are doing) but it could not be that hard to include the Figure legend on the ppt slides, even in small font. They certainly are able to inlcude their “Copyright Nature” in large font. But even if technically they are allowed to do this, they should not.

Some examples:

  • Download the powerpoint slide of Figure 1 or Figure 4 from a paper on Listeria in Nature Reviews Microbiology if you have access to it here.
  • Or look at Figure 1 of a paper in Nature Reviews Genetics here.

I guess I could load up here on other examples, but it might be more interesting for people to find their own.

Here is how I found these.

  1. Go to the Nature Advanced Search page here
  2. Search for terms like “Reproduced with permission” — I used the “The exact phrase” option.
  3. Browse away

I assume that this is not a purposeful thing they do, but it certainly seems pervasive there.
Unfortunately, it seems common in other places. For example, PNAS provides powerpoint slides but does not include Figure legends with them either. Look here. So – not trying to single out Nature here but that was where I looked first. It seems that many publishers are trying to hard to provide material (e.g., powerpoint slides) without being careful enough attributing the original sources.

Nature Journal Misappropriates Copyright of Open Access Material – though see comments …

Recently, Nature (a science journal) ran a so-called news article presenting their analysis of the finances of one of their competitors. Already I am sure one can imagine some conflicts of interest that might lead them to be really careful with such a publication. But apparently they are not as it seems to contain many flaws (see here).

Nature in this instance appears more desperate than objective, since the competitor they are criticizing is a start up society that published “Open Access” journals. Open Access means many things but one of them is that the articles are free to all. This is bad for journals like Nature that make a killing by charging people to read the results of research funded primarily by the government.

Interestingly, a little browsing around Nature’s web sites shows that not only are they apparently in a tizzy about Open Access publications, but they even have the gall to try and pretend that material published by others was generated by them.

For example, we recently published a paper on analysis of the genomes of some interesting bacteria. We published this in a PLoS Journal here.

Now take a look at this figure from the paper here.

Nature then published an article in Nature Reviews Microbiology (see here). The article is fine and even includes the figure linked above taken directly from our paper. This is OK in the world of Open Access if they attribute the origin of the figure correctly. In the article they sort of attribute it but do not do a robust job. And even more deceptively, they put “Copyright Nature” onto the Figure even though this is completely invalid. I have downloaded the figure and provide it here for those who do not have access to Nature.

I do this with no fear of the copyright gods since after all, they do not in fact have Copyrights to it.

Even worse, I saw that one can download a powerpoint slide of this figure. I did this and found that they kept the Copyright Nature part but left out the attribution so it looks like the figure is from Nature.

To me, this is plagiarism, plain and simple. And lame too.