There is a story in today’s Science magazine on Google Scholar by John Bohannon. Entitled “Google Scholar Wins Raves, But Can It Be Trusted” the article discusses some of the pros and cons of using Google Scholar. The author of the article interviewed me on and off over a few weeks about Google Scholar because I have written multiple blog posts on how I use it. For example see:
- Mini journal club: staged phage attack of a humanizes microbiome of mouse
- Weird things coming up from automated Google Scholar searches pointing to hea1thandfitness.com
- Playing with Impact Story to look at Alt Metrics for my papers, data, etc
- Thank you Google Scholar Updates for finding me two new papers to read
- Google Scholar has search by date option – a good thing
- Thanks Google Scholar Updates for new article to read .. wish you had an #openaccess only setting
- Wow – Google Scholar “Updates” a big step forward in sifting through the scientific literature
- Playing w/ Google Scholar’s “Citations” system – some nice features but in need of a little fine tuning
And also a diverse array of posts on Twitter which I will not rehash here.
Anyway – the new article covers some interesting points but is very very short (ahh — the fun with page length restrictions). So I thought I would post here some of the comments I made about Google Scholar in emails with the reporter.
Bohannon wrote to me on December 1, 2013
Dear Dr. Eisen-
I’m writing a news story for Science about Google Scholar. Have you continued using their article recommendation engine? (I saw your blog post about it from last year.)
(and then he wrote some details about what he was working on which I am not sure he would want me to post here and I have not asked so I am leaving them out)
I’d very much like to hear your thoughts on how Google Scholar has developed, and how well it works as a replacement for traditional library/proprietary/non-open literature databases.
cheers and thanks in advance,
I wrote back, that same day (unusual for me)
Well, was just hacking around with Google Scholar this AM.
Have you dug into their new function – Scholar Library? I am playing around with it but have not quite figured it out. What I am hoping to do is to figure out how to get recommendations based on lists in the Library. Currently, the recommendation engine has one very very big limitation. It bases recommendations on one’s own publications. And if you are trying to move into a new area – well that is pretty useless.
So – some comments
1. I find the recommendation engine to be very very useful still. One of the best ways to find out about new papers. With the limitation mentioned above.
2. I use automated Google Scholar searches to find all sorts of papers of interest. This helps cover topics more broadly than the recommendation engine.
3. There are other recommendation systems out there – but I have not used them too much.
4. As for “free and open” – I don’t think I would use such terminology here. Yes, Google Scholar is free. But is it open? I don’t think so. For example, I am not sure if they publish / release all their code that works behind the scenes. And I am not sure how open the results are (not saying it is not open – I just don’t know).
5. The citation information is a wonderful tool – and it is great to have this information be freely available. I use it routinely for all sorts of purposes including getting around the massive limitations of IF. One issue however with GS is that it takes citations from ALL sorts of sources including non peer reviewed material and even material that people may not have been aware was even publicly available. So – citation counts are generally higher – sometimes much higher – in GS than with other metrics. I note – I put info for my Citations based on GS on my CV and various other places.
And this allows GS to be seriously gamed in terms of citation counts.
6. GS is still clunky in a few ways and I hope that Google puts more effort into it. It could become THE tool for academic scholarship searches and tracking but it has some bugs and minor annoyances still.
Just some quick thoughts. Let me know if you have any other questions.
He then wrote back with some questions. Again, I won’t share the whole email here at this time but I will share the specific questions he asked. And I then wrote back with some answers
** Care to share a recent example? Something nicely illustrates its usefulness.
Question regarding my comment on other systems
Me: There are other recommendation systems out there – but I have not used them too much.
Bohannon: ** Are there? For example? I should take a gander.
Me: Mendeley has one – called recommended or something like thatQuestion regarding my comment on free vs. openBohannon: Right, good point. Free but not really open. Do you feel like this is a worrying limitation? Is it realistic to make it open? Or at least, more open?Me: Well, I am certainly not one to tell others how to run a business. So I am thankful for any free or open material released / produced by for profits. But there is a major worry here which is – if we do not know how the system works – we do not know if it is biased or how it can be gamed, etc. And if it is not fully open then if we invest in making use of it, Google could simply kill it at any time and we would have no source material to use for other purposes.Question regarding my comment on Impact FactorBohannon: What is an example of a massive limitation of IF that GS solves? And I wonder why GS can’t tell the difference between peer-reviewed and non-peer-reviewed sources. I’ll add that to questions below.Me: GS helps with article level metrics as opposed to journal level metrics like IF. I can look up citations to all my papers quickly. And I can track metrics like H index and I10 and others. IF – being a journal level metric – is not really that informative in my opinion (and that of many others)
Question on other things GS could do:
Bohannon: What do you see as the most important things that need to be fixed before it could really take over?
1. Transparancy in how it works and it what they are planning
2. Make it open source
3. Ability to create reference collations easily (e.g., like Mendeley or Zotero or CiteULike or Endnote).
Questions for Google
Bohannon: * Any other questions you’d like Google to answer, or features to request?
Well many features to request.
1. Better hot linking of all authors of a paper. Right now they only seem to link the 1st author to their google scholar page.
2. Better handling of long author lists.
3. Better ability to upload collections exported from other tools.
And many more …
After this Bohannon then wrote me another email with additional questions which I answered, briefly
Bohannon: Questions I should have asked but forgot:Bohannon: When was the last time you exclusively used paper journals to find articles? (Ever?)Answer from me:never – or a very very very long time agosince I started using email / the web 20+ years ago I have tried to use electronic / digital searches to find papersBohannon: What electronic services did you use before GS?Answer from me:Pubmed searches of courseAnd pubmed had some automated email alerts that I used to useI also used lots of searches of journal web sitesAnd some journals offered automated searches tooThe best thing out there was probably “related articles” in PubmedBohannon: Have you switched entirely to GS?Answer from me:No – I still use pubmed searches quite a bit – partly because they are less cluttered than GS searchesI still occasionally search journal web sites but that is rareAnd I actually use straight google searches a lot
And then another question
Bohannon: Oh, and when did you start using GS? And when did it become your main search service?
(Sorry for the shotgun interrogation!)
Answer 1 from me:
heh – I have no idea …
Answer 2 from me:
I note – I think right now Twitter is the best source of information about new papers — far better than GS …
And I looked through my email and found that for a few years I used F1000 automated searches
And then in response to a follow up comment about Twitter from Bohannon I said
Yes, did not say Twitter was fun … but if you follow the right people, they can crowdsource for you all of the right literature better than GS
And then on Dec 6 Bohannon wrote me about visualizations
Hi Jonathan-We’re sorting out some art for the article about Google Scholar. I’m thinking that the best option would be a cropped screen shot of one of your Google Scholar automatic article recommendations. Would you be OK with that?Do they come as an email or appear in browser? If email, you can just fwd it. If browser, could you take a screen shot and send it?The idea is to show people what GS article recommendations look like, using your own experience for a visual.
So I sent a few things including these
We had a few other discussions about related topics but I thought it might be useful to see that general gist of the discussion and some of my thoughts on Google Scholar in more detail. I note, at our meeting in February on the Future of Scholarly Publishing and Careers here at UC Davis Anurag Acharya from Google Scholar will be talking and will be on a discussion panel that will focus broadly on “metrics” for scholars so should be interesting.
One thought on “Some additional details of my discussion w/ reporter John Bohannon for his Science story on Google Scholar”
Just a thought about having the algorithm open-source – would that make it easier to game the system or harder (because the crowd would strengthen the code)?
At least with a closed source it is a less trivial task to corrupt the citations.
However, as you point out if Google decides to abandon Scholar without the source code we are left in the lurch.