Well, I guess I can say I was not pleased to see this tweet from Carl Zimmer.
Oh for crying out loud. http://t.co/MjekLb3Av5
— carlzimmer (@carlzimmer) February 1, 2014
//platform.twitter.com/widgets.js It is not that I have a problem with what Carl wrote. It is just that, then I went and read the article he referred to: Scientific Pride and Prejudice in the New York Times By Michael Suk-Young Chwe. And it just did not make me happy. I reread it. Again and again. And I was still unhappy.
What bugs me about this article? Well, alas, a lot. The general gist of the article is that “natural” scientists are not aware enough of how their own preconceptions might bias their work. And furthermore that literary criticism is the place to look for such self-awareness. Well, interesting idea I guess but alas, the irony is, this essay presents no evidence that literary criticism does better with evidence than natural science. Below are some of the lines / comments in the article that I am skeptical of:
- “Scientists now worry that many published scientific results are simply not true.”
- This implies that before the Nature article from two years ago that is references here that scientists did not worry that many published results were not true. This is simply untrue.
- Here are some links to papers discussing lack of reproducibility and ignoring confirmation bias as an issue in Science from before two years ago. There are I note – hundreds of these.
- Problems of reproducibility–does geologically ancient DNA survive in amber–preserved insects?
- Effect of interpretive bias on research evidence
- Hypothesis testing in ecology: psychological aspects and the importance of theory maturation
- Seeing what we want to see: Confirmation bias in animal behavior research
- Objective assessment of scientific performances world-wide
- Uncovering negative results: Introducing an open access journal “Journal of Pharmaceutical Negative Results”
- Estimating the proportion of studies missing for meta-analysis due to publication bias
- Negative changes in the scientific publication process in ecology: potential causes and consequences
- Research bias: Some preliminary findings
- Now – I am not saying everyone in all the natural sciences thinks about issues like confirmation bias and reproducibility. But to imply that people only started thinking about this due to a Nature paper from two years ago is not only without evidence, it runs against the available evidence which is easy to find.
- “Scientists, eager to make striking new claims, focus only on evidence that supports their preconceptions. Psychologists call this “confirmation bias. We seek out information that confirms what we already believe. ”
- This statement is misleading. Confirmation bias according to all definitions I could find is something more subtle. For example Oxford Dictionaries defines it as “the tendency to interpret new evidence as confirmation of one’s existing beliefs or theories.” That is, it is a tendency – a leaning – a bias of sorts.
- I would very much like to see evidence behind the much more extreme claim of this author that scientists focus “only on evidence that supports their preconceptions”.
- In my readings of actual research on confirmation bias I can find no evidence to this claim. For example see the following paper Confirmation bias: a ubiquitous phenomenon in many guises. which states:
- As the term is used in this article and, I believe, generally by psychologists, confirmation bias connotes a less explicit, less consciously one-sided case-building process. It refers usually to unwitting selectivity in the acquisition and use of evidence. The line between deliberate selectivity in the use of evidence and unwitting molding of facts to fit hypotheses or beliefs is a difficult one to draw in practice, but the distinction is meaningful conceptually, and confirmation bias has more to do with the latter than with the former.
- “Despite the popular belief that anything goes in literary criticism, the field has real standards of scholarly validity”
- This is a red herring to me. I can find no evidence that
theirthere is a popular belief that “anything goes” in literary criticism. So the author here sets a very low bar and then basically any presentation of standards is supposed to impress us. - “Rather, “the important thing is to be aware of one’s own bias.”
- The author then goes on to discuss how those in the humanities are aware of the issues of confirmation bias and rather than trying to get rid of it, they just deal with it, as implied in the quote.
- The author then writes “To deal with the problem of selective use of data, the scientific community must become self-aware and realize that it has a problem. In literary criticism, the question of how one’s arguments are influenced by one’s prejudgments has been a central methodological issue for decades.“
- Again, this implies that scientists have not been thinking about this at all which is just wrong.
- And then the author uses the Arsenic-life story as an example of how scientists suffer from “confirmation bias.” If you do not know about the arsenic life story see here. What is the evidence that this was “confirmation bias“?. I think more likely this was a case of purposeful misleading, overhyping, and bad science.
- Then the author gives as an example of how science actually is prone to confirmation bias by presenting a discussion of Robert Millikan’s notebooks in relation to a classic “oil drop” experiment. Apparently, these notebooks show that the experiments got better and better over time and closer to the truth. And in the notebooks Millikan annotated them with things like “Best yet – Beauty – Publish”. And then the author concludes this means “In other words, Millikan excluded the data that seemed erroneous and included data that he liked, embracing his own confirmation bias.” I don’t see evidence that this is confirmation bias. I think better examples of confirmation bias would be cases where we have now concluded the research conclusions were wrong. But instead, Millikan was and still is as far as I know, considered to have been correct. He won the Nobel Prize in 1923 for his work. Yes, there has been some criticism of his work but as far as I can tell, there is no evidence that he had confirmation bias.
- I am going to skip commenting on the game theory claims in this article.
- Then the author writes “Perhaps because of its self-awareness about what Austen would call the “whims and caprices” of human reasoning, the field of psychology has been most aggressive in dealing with doubts about the validity of its research.” Again – what is the evidence for this? Is there any evidence that the field of psychology is somehow different?
I could go on and on. But I won’t. I will just say one thing. I find it disappointing and incredibly ironic that an article that makes claims about how some fields deal better with evidence and conformation bias than other fields does not present any actual evidence to back up its claims. And many of the claims pretty clearly run counter to available evidence.
UPDATE 9:20 AM 2/2/2014: Storify of discussions on Twitter
//storify.com/phylogenomics/discussion-of-scientific-pride-and-prejudice-and-m/embed//storify.com/phylogenomics/discussion-of-scientific-pride-and-prejudice-and-m.js[View the story “Discussion of “Scientific Pride and Prejudice” and my response to it” on Storify]
Jonathan Eisen's Lab 