Tag: Science Magazine

Storify of Discussions About Shameless Behavior by #AAAS Relating to #Ebola Papers

//storify.com/phylogenomics/aaas-science-magazine-shamelessly-try-to-get-credi/embed?border=false//storify.com/phylogenomics/aaas-science-magazine-shamelessly-try-to-get-credi.js?border=false[View the story “#AAAS & Science Magazine shamelessly try to get credit for free #Ebola papers; while ignoring HIV, TB, malaria, etc ” on Storify]
——–
This is from the “Tree of Life Blog”
of Jonathan Eisen, an evolutionary biologist and Open Access advocate
at the University of California, Davis. For short updates, follow me on Twitter.

——–

Please make it stop – overselling the microbiome award for rugby, exercise, microbiome stories

Update added 11/2/14 – for all my posts on Overselling the Microbiome go here. 

Well, I think today’s lesson is, many people, including many scientists and science reporters, just do not get that there is a difference between correlation and causation.  I know – this is like beating a dead horse since many write about this issue.  But it just needs to be called out every time until it stops.  And today’s fun comes from stories and the original research articles about how exercise supposedly alters the gut microbiome.

I was pointed to this just a few minutes ago on Twitter:

//platform.twitter.com/widgets.js
In this Tweet Bernat Olle points to a “news” story in Medpage Today: Exercise Boosts Gut Microbiome Diversity by Kristina Fiore.   Well, so of course I started digging around.  And, not surprisingly, the study that this is based on shows absolutely no causal connection between exercise and the gut microbiome.  The study is in the journal “Gut”: Exercise and associated dietary extremes impact on gut microbial diversity.  And here is what they did:

  • They selected subjects – 40 “elite” rugby players.
  • They identified healthy male “controls” with similar age and size and from similar place. 
  • Then they collected faecal and blood samples from participants and did surveys about their nutrition and clinical data.
  • Among many measurements, they did 16S sequencing from the fecal samples
  • Then they did some bioinformatics and found differences between the rugby players and the controls in many features including microbiomes.
And amazingly, from this they report, in their abstract

The results provide evidence for a beneficial impact of exercise on gut microbiota diversity but also indicate that the relationship is complex and is related to accompanying dietary extremes.

The key part of this to me is 

The results provide evidence for a beneficial impact of exercise on gut microbiota diversity

For which they have no support.  They do not in any way show that exercise has ANY affect on the microbiota.  They show it is correlated to the microbiota.

And sadly there is a commentary on the article in the same issue of Gut that makes the same mistake.  Georgina Hold in The gut microbiota, dietary extremes and exercise writes:

The article is the first report that exercise increases gut microbiota richness/diversity and highlights that exercise is another important factor in the complex relationship among the host, host immunity and the microbiota.

No.  They did not show exercise increased gut microbiota diversity.  How can the difference between correlation and causation be missed in these articles?  Are these not even reviewed?  Sure – this is consistent with exercise affecting microbiomes but it is also consistent with rugby players having different diets and other behaviors.  There is a big difference between showing cause and effect and showing correlation.  For not distinguishing between correlation and causation regarding the rugby player microbiomes I am giving all involved here an “Overselling the Microbiome Award“.

Here is a microbiome theory I will leave you with.  I hypothesize that these papers, and all the other ones that oversell the microbiome, themselves cause major changes in the microbiome of many people.  Evidence for this?  Well, none yet.  But I have a correlation.  The correlation is, after reading these papers,  I feel sick to my stomach.  That must be proof right?

UPDATE 6/11/14

Author of the Medscape Medpage today article Kristina Fiore says she will update the article to more accurately reflect the science. See some of the thread below

//platform.twitter.com/widgets.js

//platform.twitter.com/widgets.js

//platform.twitter.com/widgets.js

UPDATE 2: 6/11/14.

The press release from Gut associated with this paper contains many inaccurate statements.

Examples include:

  • Title: Exercise boosts diversity of gut bacteria
  • Text: Exercise boosts the diversity of the bacteria found in the gut, indicates the first study of its kind published online in the journal Gut.
Somewhat surprised that such mistakes would come from the journal itself.

UPDATE 3: 6/12/14.

Kritina Fiore has fixed the Medpage article.  Nice.

UPDATE 4: 6/12/14.

Science Magazine gets the causation vs. correlation issue wrong in their little news piece about this.  Yuck.

//platform.twitter.com/widgets.js

UPDATE 5: 6/12.

Alexandra Sifferlin has a good article about this at Time

//platform.twitter.com/widgets.js

//platform.twitter.com/widgets.js

UPDATE 6.

More accurate coverage by Claire O’Connell in the Irish Times Generally a good article here: Rugby players show good guts

UPDATE 7.

Popular Science messes it up too

UPDATE 8.

Keeping track of some of the Tweets about this on Storfy.

UPDATE 9 6/13/14.

NPR News Falls for the Hype

//platform.twitter.com/widgets.js

UPDATE 10: 6/13/14.

Just found another inaccurate claim in the original paper

UPDATE 11: 6/13/14

Oh FFS. Now I have found some articles reporting not only that exercise affects gut microbial diversity but that this is why exercise reduces obesity. See Exercise lowers obesity risk by stimulating diverse gut bacteria in the NVO News, for example.

 

Quote from the story:

A latest research suggests that exercise actually lowers obesity risk by stimulating diverse gut bacteria

UPDATE 12: 6/13/14

Fox News did better with the science (at least in their headline) than many other News Agencies (and much better than NPR).  They report “Exercise may lead to healthier gut bacteria“.

Just that word “May” makes me happy.  I know.  Low bar.  But I will take what I can get.

UPDATE 13: 6/13/14

Genome Web also is reporting on the story and on the “overselling” that was done.

UPDATE 13: 6/18/14

And now the New York Times joins the fray: Exercise and the ‘Good’ Bugs in Our Gut where Gretchen Reynolds writes:

The findings suggest that, in addition to its other health benefits, frequent exercise may influence our weight and overall health by altering the kinds of organisms that live inside of us.

No – the findings do not suggest that.  The findings are consistent with that theory but they are consistent with many many many other theories.  FFS this is maddening.  And the article ends with a quote from one of the authors:

But even in advance of those findings, he said, it seems likely that any amount of exercise should make your gut more welcoming to the bacteria that you want residing there.

I note – I found out about this article via Twitter

//platform.twitter.com/widgets.js

——–
This is from the “Tree of Life Blog”
of Jonathan Eisen, an evolutionary biologist and Open Access advocate
at the University of California, Davis. For short updates, follow me on Twitter.

——–

Some additional details of my discussion w/ reporter John Bohannon for his Science story on Google Scholar

There is a story in today’s Science magazine on Google Scholar by John Bohannon.  Entitled “Google Scholar Wins Raves, But Can It Be Trusted” the article discusses some of the pros and cons of using Google Scholar.  The author of the article interviewed me on and off over a few weeks about Google Scholar because I have written multiple blog posts on how I use it.  For example see:

And also a diverse array of posts on Twitter which I will not rehash here.

Anyway – the new article covers some interesting points but is very very short (ahh — the fun with page length restrictions).  So I thought I would post here some of the comments I made about Google Scholar in emails with the reporter.

Bohannon wrote to me on December 1, 2013

Dear Dr. Eisen-

I’m writing a news story for Science about Google Scholar. Have you continued using their article recommendation engine? (I saw your blog post about it from last year.)

(and then he wrote some details about what he was working on which I am not sure he would want me to post here and I have not asked so I am leaving them out)

I’d very much like to hear your thoughts on how Google Scholar has developed, and how well it works as a replacement for traditional library/proprietary/non-open literature databases.

cheers and thanks in advance,
John Bohannon

I wrote back, that same day (unusual for me)

Well, was just hacking around with Google Scholar this AM.

Have you dug into their new function – Scholar Library? I am playing around with it but have not quite figured it out. What I am hoping to do is to figure out how to get recommendations based on lists in the Library. Currently, the recommendation engine has one very very big limitation. It bases recommendations on one’s own publications. And if you are trying to move into a new area – well that is pretty useless.

So – some comments

1. I find the recommendation engine to be very very useful still. One of the best ways to find out about new papers. With the limitation mentioned above.

2. I use automated Google Scholar searches to find all sorts of papers of interest. This helps cover topics more broadly than the recommendation engine.

3. There are other recommendation systems out there – but I have not used them too much.

4. As for “free and open” – I don’t think I would use such terminology here. Yes, Google Scholar is free. But is it open? I don’t think so. For example, I am not sure if they publish / release all their code that works behind the scenes. And I am not sure how open the results are (not saying it is not open – I just don’t know).

5. The citation information is a wonderful tool – and it is great to have this information be freely available. I use it routinely for all sorts of purposes including getting around the massive limitations of IF. One issue however with GS is that it takes citations from ALL sorts of sources including non peer reviewed material and even material that people may not have been aware was even publicly available. So – citation counts are generally higher – sometimes much higher – in GS than with other metrics. I note – I put info for my Citations based on GS on my CV and various other places.

And this allows GS to be seriously gamed in terms of citation counts.

6. GS is still clunky in a few ways and I hope that Google puts more effort into it. It could become THE tool for academic scholarship searches and tracking but it has some bugs and minor annoyances still.

Just some quick thoughts. Let me know if you have any other questions.

Jonathan Eisen

 

He then wrote back with some questions.  Again, I won’t share the whole email here at this time but I will share the specific questions he asked.  And I then wrote back with some answers

Question:

** Care to share a recent example? Something nicely illustrates its usefulness.

Answer:

 

Question regarding my comment on other systems

Me: There are other recommendation systems out there –  but I have not used them too much.

Bohannon: ** Are there?  For example?  I should take a gander.

Me: Mendeley has one – called recommended or something like that

Question regarding my comment on free vs. open
Bohannon: Right, good point. Free but not really open. Do you feel like this is a worrying limitation? Is it realistic to make it open?  Or at least, more open?
Me: Well, I am certainly not one to tell others how to run a business.  So I am thankful for any free or open material released / produced by for profits.  But there is a major worry here which is – if we do not know how the system works – we do not know if it is biased or how it can be gamed, etc.  And if it is not fully open then if we invest in making use of it, Google could simply kill it at any time and we would have no source material to use for other purposes.
Question regarding my comment on Impact Factor
Bohannon: What is an example of a massive limitation of IF that GS solves? And I wonder why GS can’t tell the difference between peer-reviewed and non-peer-reviewed sources. I’ll add that to questions below.
Me: GS helps with article level metrics as opposed to journal level metrics like IF.  I can look up citations to all my papers quickly.  And I can track metrics like H index and I10 and others.  IF – being a journal level metric – is not really that informative in my opinion (and that of many others)

Question on other things GS could do:

Bohannon: What do you see as the most important things that need to be fixed before it could really take over?

Me:

1. Transparancy in how it works and it what they are planning
2. Make it open source
3. Ability to create reference collations easily (e.g., like Mendeley or Zotero or CiteULike or Endnote).

Questions for Google

Bohannon: * Any other questions you’d like Google to answer, or features to request?

Me:

Well many features to request.

1. Better hot linking of all authors of a paper. Right now they only seem to link the 1st author to their google scholar page.

2. Better handling of long author lists.

3. Better ability to upload collections exported from other tools.

And many more …

After this Bohannon then wrote me another email with additional questions which I answered, briefly

Bohannon: Questions I should have asked but forgot:
Bohannon: When was the last time you exclusively used paper journals to find articles? (Ever?)
Answer from me:
never – or a very very very long time ago
since I started using email / the web 20+ years ago I have tried to use electronic / digital searches to find papers
Bohannon: What electronic services did you use before GS?
Answer from me:
Pubmed searches of course
And pubmed had some automated email alerts that I used to use
I also used lots of searches of journal web sites
And some journals offered automated searches too
The best thing out there was probably “related articles” in Pubmed
Bohannon: Have you switched entirely to GS?
Answer from me:
No – I still use pubmed searches quite a bit – partly because they are less cluttered than GS searches
I still occasionally search journal web sites but that is rare
And I actually use straight google searches a lot

And then another question

Bohannon: Oh, and when did you start using GS?  And when did it become your main search service?

(Sorry for the shotgun interrogation!)

Answer 1 from me:

heh – I have no idea …

Answer 2 from me:

I note – I think right now Twitter is the best source of information about new papers — far better than GS …

And I looked through my email and found that for a few years I used F1000 automated searches

And then in response to a follow up comment about Twitter from Bohannon I said

Yes, did not say Twitter was fun … but if you follow the right people, they can crowdsource for you all of the right literature better than GS

 

And then on Dec 6 Bohannon wrote me about visualizations

Hi Jonathan-

We’re sorting out some art for the article about Google Scholar.  I’m thinking that the best option would be a cropped screen shot of one of your Google Scholar automatic article recommendations.  Would you be OK with that?
Do they come as an email or appear in browser? If email, you can just fwd it.  If browser, could you take a screen shot and send it?
The idea is to show people what GS article recommendations look like, using your own experience for a visual.

 

So I sent a few things including these

 

Screen Shot 2013-12-06 at 12.09.09 PM

 

Screen Shot 2013-12-06 at 12.05.40 PM

We had a few other discussions about related topics but I thought it might be useful to see that general gist of the discussion and some of my thoughts on Google Scholar in more detail.  I note, at our meeting in February on the Future of Scholarly Publishing and Careers here at UC Davis Anurag Acharya from Google Scholar will be talking and will be on a discussion panel that will focus broadly on “metrics” for scholars so should be interesting.