guest post – Jonathan Eisen's Lab

Blind trust in unblinded observation in Ecology, Evolution and Behavior (Guest Post by Melissa Kardish)

This is a guest post from Melissa Kardish – a PhD student at UC Davis – writing about a recent paper from work she did at her prior position. The citation for the paper she is writing about is below:

Kardish MR, Mueller UG, Amador-Vargas S, Dietrich EI, Ma R, Barrett B and Fang C-C (2015) Blind trust in unblinded observation in Ecology, Evolution, and Behavior. Front. Ecol. Evol. 3:51. doi: 10.3389/fevo.2015.00051

Here is her post.

Blind trust in unblinded observation in Ecology, Evolution and Behavior

We recently published our study in Frontiers in Ecology and Evolution where we found that a remarkable number of studies that could be affected by observer bias didn’t indicate whether or not they blinded their research. In fact only 13.3% of studies reported this:

We tried to make this a very transparent study. In addition to journal level data in the main article, we include in our supplemental material a table with the score for every article we read for this study (a summary of these scores per journal can be found in Figure S2 included here). If anything, our results under-represent the amount of studies that could have been scored blind (the real underreporting/underuse of blind observation is probably less than the 13.3% we report). For instance, we did not assess that there was potential for bias in the scoring of microsatellite markers (scored as unlikely to have observer bias). However, we did identify one study which was based on data from microsatellites which did blindly score their markers and report this scoring in their methods (and was therefore scored as “blind” in our study). We also considered a study blind in its entirety for the purposes of our scoring if only one aspect is reported even if other experiments could also have been influenced by observer bias (Check out our supplemental methods for more ways we conservatively scored in our study).

We recognize that not all EEB studies can be blinded due to a variety of logistical or hypothesis driven reasons; however, we encourage such studies to accurately report this rationale and consider and attempt to minimize observer bias when designing experiments.

Thus far we have had a great response from the surveyed journals. Many of them have notified their editors about the lack of blind observation that we found reported in their journal. One journal has even notified us of plans already in place to address this issue at their next editorial board meeting.

We’re excited to have this work out there and hope this will inspire people to blind their studies and accurately report the science they are doing. We’re also excited to have the study published in an open-access format where we hope the encouragement for blind observation can reach all levels of science. Finally, as reporting of science in our fields improves in the coming years, we hope this study can serve as a template to address other potential concerns in experimental design and reporting.

For our article check out: http://journal.frontiersin.org/article/10.3389/fevo.2015.00051/full

For our supplemental text and methods: http://journal.frontiersin.org/file/downloadfile/10238/octet-stream/Data%20Sheet%201.PDF/711/2/142945

For our data table with each of the articles we surveyed: http://journal.frontiersin.org/file/downloadfile/10235/octet-stream/Table%201.XLSX/711/1/142945

Guest post from Jake Scott: Building trust: a sine qua non for successful acceptance of preprints in the biological sciences

Today I am happy to have a guest post from my friend and colleague Jake Scott. The topic of the day is preprints in biology and medicine.

Hi – I’m Jake Scott. I met Jonathan last year when he and I spoke at TEDMED 2012. Both Jonathan and I have posted recently about the need for, and (slowly) growing movement in the biological sciences to post #preprints of manuscripts in openly accessible fora to circumvent some problems associated with standard academic publishing. Most worrisome are the issues surrounding #openaccess and the length of time it takes to get information from one’s brain to the literature – drastically slowing down the pace of science.

This has worked GREAT in the physics community, where this trend really began quite some time ago when the high energy physicists started the arXiv. Now, the precedent is set, and no one in physics bats an eye about sticking their paper on the arXiv, and cite other works presented there as standard publications.

The climate in biology, sadly, is much different. Whether this is because of a more competitive climate for funding, or just a field diluted by more talented scientists, I don’t know. But there is a pervasive attitude of fear and mistrust around the idea of preprints.

Before you read on (and become biased by my opinions) take a few second (really, probably 1.5 minutes) and take this quick survey:

When I preach to my biological colleagues about the virtue of pre-print servers, I most often, I hear:

Why should I post my papers on a pre-print server where anyone can see it before it is published!? They could scoop me!

I honestly don’t understand this argument, but I hear it all the time. By nature of pre-print servers, like the arXiv, the idea is yours! Time and date stamped. And, better yet, it is completely #openaccess, free of charge, and helps move science along at a better pace. Only a very few journals have problems with posting of pre-prints before they get their (greedy) hands on the results of all your hard work, but most are totally OK with it.

The arXiv isn’t really interested in shopping its (free) service out to the biological sciences, not because they don’t think it would be of value, but because it just doesn’t have the infrastructure to support it. This is a problem that is being with newly created repositories like Nature Precedings, PeerJ and soon, the bioRxiv. So, the only thing holding us up is, IMHO, trust.

How can we rectify this?

I think the way forward is to create something that we are all missing now, except when we are at our home conference, among friends or if we got into a time machine and went back 100 years – community.

Science is such a juggernaut now that putting your work onto a pre-print server where anyone in the world can see your as of yet unvetted work can be daunting. Worse, the idea of commenting on it is a tough sell when the world is a witness. I think we need to (re)create micro-communities of our specialist peers where these initial discussions can be held. Two examples of this are Haldane’s Sieve and more recently created, an initiative I’m involved with, Warburg’s Lens. These two sites are micro-communities where population and evolutionary biologists, and mathematical oncologists (respectively) congregate to discuss pre-prints culled from any repository but necessarily of interest to the micro-community.

This does two things: it allows a common place for easy browsing in topics of interest to a specialist (like reading your favorite journal), and increases the chances that the readers and commenters are your (at large) peers.

So, those are my two cents. #Openaccess for all is coming, and preprints are a part of the wave. The sooner we all adopt an open science attitude, the sooner we’ll come to the conclusions and make the discoveries that make doing science AWESOME. There is no better job than science, and sharing and communication are central to it

So START SHARING your science. Commit to this – when you are ready to submit your next paper, put that version on a pre-print server as you start the submission process. Then tweet about it, G+ about it, blog about it, do whatever, but let your peers know!

Anyone else interested in starting a micro-community discussion forum, or to just discuss this issue further, please contact me.

If you are against it – please leave some comments about why, I’d love to try to convince you otherwise! If you are a biologist (or know one) who DOES post pre-prints, weigh in and share your good experiences!

About me: I am a radiation oncologist and I approach the understanding of cancer like my original training in physics taught me – from the ground up, using the descriptive language of mathematics. Using established mathematics in new ways, guided by the principles of evolution, I hope to better understand (and maybe treat!) cancer. I am a proud member of the Integrated Mathematical Oncology group at the Moffitt Cancer Center and the Centre for Mathematical Biology at Oxford University. You can follow me on twitter @CancerConnector or read my blog Connecting the Dots.

Guest post from Katherine Scott of the Journal of Visualized Experiments on #OpenAccess challenges

Today we have another guest post here. This one is from Katherine Scott from JOVE – the Journal of Visualized Experiments. I really like the concept behind JOVE – high quality videos of experimental protocols. Publications in JOVE were initially freely available to all (see my 2008 post about JOVE here). Alas, a few years ago, things changed with the introduction of a subscription model. This saddened many out there, myself included, since JOVE was a wonderful addition to the collection of freely available scientific resources. I wish they had been able to avoid this, but it seems that they could not. Katherine Scott from JOVE explains their side of the story below:

Guest post by Katherine Scott “Open Access from the Perspective of an Academic Journal”

Open access from the perspective of an academic journal. I work for the first and only peer-reviewed science video journal indexed in PubMed and MEDLINE, the Journal of Visualized Experiments (JoVE). We started as an open access resource in 2006, but that model wasn’t sustainable for us. The cost of producing high-quality video simply too high.

So how do we remain profitable without losing our open access roots? Balance.

We started offering subscriptions in 2009, but still try to open up access wherever we can. We recently partnered with Health InterNetwork Access to Research Initiative (HINARI), to give free subscriptions to developing countries in South America, Asia and Africa.

HINARI, a World Health Organization (WHO) initiative, grants developing countries access to one of the largest collections of biomedical and health literature. It was founded in 2002 after a WHO survey found that 56 percent of institutions in the poorest countries had no current subscriptions to academic journals.

“Researchers from developing countries were saying ‘we need access to subscription literature, we can’t afford it, and without it, we can’t be part of the global research community,” said HINARI Library Program Manager Kimberly Parker.

Despite now having a large body of literature available to them, Parker said that students and researchers were still struggling because of language barriers and little access to proper demonstrations of experimental techniques in labs. She believes the visual aspect of JoVE will help address those problems.

Visual demonstrations of experimental techniques is the reason Dr. Lucia Prieto Godino, a post-doc at Cambridge University, asked for permission to use JoVE for the Drodophila Neurogenetics course she is teaching at Kampala International University in Uganda.

“With the JoVE articles they will be able to see the whole protocol, taught by an expert,” said Dr. Godino. “For them, JoVE is particularly important because they can’t pop by another lab to find an expert and learn.”

Now that HINARI will be carrying JoVE videos, the students will not only be able to see the experiments during her course, they will also be able to watch them again at their home institutions.

As much as it may break our hearts that we can’t survive as a purely open access resource to everyone, it’s great to know that subscriptions make it possible for us to provide experimental videos to those who need them most.