Confirmation Bias

Adam Shostack has a couple of posts on Confirmation Bias. I've added some comments on Adam's blog; here's a digest of the discussion.

Things Only An Astrologist Could Believe

Adam picks up an astrological analysis of a recent action by Google: apparently the timing of the Chrome release was astrologically auspicious.

Vedic Astrologer: "Such a choice of excellent Muhurta with Chrome release time may be coincidental, but it makes us strongly believe that Google may not have hesitated to utilize the valuable knowledge available in Vedic Astrology in decision making."

Adam: "This is a beautiful example of confirmation bias at work. Confirmation bias is when you believe something (say, Vedic astrology) and go looking for confirmation. This doesn't advance your knowledge in any way. You need to look for contradictory evidence. For example, if you think Google is using Vedic astrology, they have a decade of product launches with some obvious successes. Test the idea. I strongly believe that you haven't."

Myself: "What our Vedic friend is actually telling us is that Google "may not have hesitated" in its use of Vedic astrology. To be honest, I also find it hard to believe that Google executives sat around dithering about whether to use Vedic astrology or not."

In further comments, the Vedic astrologer argues that the astrological method is no different from other forms of observational science, using the scientific method, which requires the prediction of future results.

• Hypothesis: The sun comes up every 24 hrs.
• Method: I will time when the sun crosses the horizon.
• Results: I successfully predicted 50 sunrises with a 100% degree of accuracy. This is further evidence that my hypothesis is correct.
• Caveat: Although, I note that since 24 hrs is the period between sunrises by definition of a day, this is circular.

Actually, the statement that the sun rises exactly in 24 hour intervals is only believable if you live near the equator and you know nothing about astronomy, or if you adopt a solar method for measuring the length of an hour.

What confuses me about the hypothesis posed by our Vedic friend is whether he is trying to predict the decision-making behaviour of Google executives or the successful outcome of their decisions. Even if Google executives are making auspicious decisions, this could be "explained" either by the fact that they are employing the services of an astrologer, or by the fact that Google happens to have good (= astrologically blessed) executives. Or something.

More on Confirmation Bias

According to an old article by Michael Shermer in the Scientific American [The Political Brain, June 2006], "a recent brain-imaging study shows that our political predilections are a product of unconscious confirmation bias". Devan Desai concludes that "hardcore left-wing and hardcore right-wing folks don’t process new data".

When I first read that line about "hardcore left-wing and hardcore right-wing folks" quoted in Adam's blog I assumed it was talking about serious extremists - communists and neoNazis. Turns out it was just looking at people with strong Democrat or Republican affiliation. Maybe any party affiliation at all seems pretty hardcore to some people.

As far as I can see, the study only actually looked at people with strong political opinions, and didn't compare them with any control group. Like, er, the middle-of-the-road folks who fund and write up this kind of research.

I wonder whether anyone would get research funding or wide publicity for exploring the converse hypothesis - that people with strong political opinions are actually relatively open-minded, and that the people who have the most entrenched opinions are the bureaucrats who staff the research funding bodies and the people who write popular articles for Scientific American.

(Of course I'm jumping to conclusions myself here, that's what bloggers do isn't it?)

I'm not saying I believe that bigots are more open-minded than wishy-washy middle-of-the-roaders. I'm just saying we need to be mistrustful of studies that are designed to confirm the prejudices of the researchers, and suspicious of people who latch onto these studies to prove a point. The problem is that there may be confirmation bias built into the way these kind of pseudo-scientific studies are funded, organized and then publicized. Not surprising then if "the FMRI findings merely demonstrate what many of us already knew".

As director of the Skeptics Society, Michael Shermer latches onto a study showing that people are biased. Shermer himself has a particular set of bugbears, including evolutionary psychology, alien abductions, prayer and healing, and alternative medicine. Are we really to imagine that he approaches any of these topics with a truly open mind? And why should he anyway? The rest of us don't.

Science and Censorship

Dark materials. Nuclear scientist Joseph Rotblat campaigned against the atom bomb he had helped unleash. Is it time for today's cyber scientists to heed his legacy?

"There is an ever-widening gap between what science allows, and what we should actually do. There are many doors science can open that should be kept closed, on prudential or ethical grounds. Choices on how science is applied should not be made just by scientists."

Essay by Martin Rees
President of the Royal Society
[Guardian, Saturday June 10, 2006]

---

We cannot allow the terrorists to terrorise us. Scientific research shouldn't be halted simply because it might fall into the wrong hands.

"The scientist's job is to shine light in the darkness, and if we occasionally burn our fingers on the candle, so be it. Lord Rees can choose the darkness if he wants. I'm not going to."

Response by Ross Anderson
Chair of the Foundation for Information Policy Research
[Guardian, Tuesday June 20, 2006]

---

I am uneasy about both sides of this debate. Should science be restrained - either by scientists or by society. Do politicians represent the interests of society, or is there a better and more democratic way for society's interests to be represented?

The fact is that science is already restrained by all sorts of social and commercial forces - above all the willingness to fund particular kinds of research and not others. The choice is not simply between a risk-averse establishment (represented by the Royal Society) and a risk-seeking free-thinking radical alternative (represented by the Foundation).

Of course Anderson is right to be wary of the distorted perceptions of risk by politicians and the non-scientific public. But the proper response to this is a properly constituted debate. Meanwhile, politicians will often seek stupid measures to use and abuse scientists.

As I reported last year (Research Under Fire), scientists and engineers at the University of Berkeley are wary of academic restrictions imposed by the US Federal Government in the name of national security. Thankfully this isn't the kind of restraint Rees is advocating.

Technorati Tags: censorship research science

Conflicting Opinions

Disagreements are unsettling. In a stable world, we like our experts to provide simple and authoritative truth.

Science isn't really supposed to work like that. Science should be constantly open to new discoveries, and new interpretations and explanations of old discoveries. [Kant, Peirce] Science is supposed to work with conjectures, proofs, refutations and paradigm shifts. [Popper, Kuhn, Lakatos, Feyerabend] These are essentially architectural notions. [See my earlier post on Open Architecture]

But the institutions and bureaucracies of science don't conform to this stereotype. There is a closed loop of research funding and journal publication, based on so-called peer review. In a recent report the Royal Society, Britain's most prestigious scientific club, deplores the popular media coverage of science and calls upon scientists to exercise greater self-restraint in publicizing findings that have not undergone the proper peer review process. [Source: John Kay, Financial Times, May 22, 2006]

As John Kay argues, this is essentially an appeal for scientists to be dull and boring. Peer review is inward looking, inhibits new and radical ideas, and serves as a kind of professional censorship. He concludes

"Any form of censorship, including self-censorship and censorship by fellow professionals, encourages complacency and discourages innovation. The history of modern scholarship is that, more slowly than we would wish, truth and new knowledge emerge only from a cacophony of conflicting opinions."

Technorati Tags: architecture Kant knowledge open Peirce uncertainty

Dilbert on Intelligent Design

Interesting post by Dilbert on Intelligent Design, which raises some interesting questions of Knowledge and Trust.

He makes the following points.

1. The arguments for Darwinism (and intellectual defences against the "flaws" identified by the Intelligent Design and Creation folk) are complex.

2. Belief in Darwinism depends on trusting the vast majority of scientists working in the field.

3. However, the scientific field relevant to Darwinism is compartmentalized. Scientists are required to trust evidence from other specialisms and disciplines. We are not just talking about non-scientists trusting scientists, but scientists trusting other scientists.

4. Therefore the entire scientific edifice of Darwinism is based on inter-disciplinary trust.

5. If the institution of science is anything like the organizations that Dilbert has made a fortune analysing and drawing, then we have to take seriously the possibility that they've all got it completely wrong.

Of course, the possibility that science has got things completely wrong has been explored by philosophers of science such as Kuhn, Lakatos and Feyerabend. But Dilbert is pointing to a new angle on this - the institutional mechanisms (familiar in large organizations) that permit lots of small bits of evidence to be accumulated and amplified into false knowledge.

Dilbert is making a profound point about the way knowledge is composed from lots of bits of evidence. If much of this evidence is stronger when seen from a distance, and weaker when examined closely, this seems to call the whole body of knowledge into question. (Dilbert doesn't go into the recursive loops of analysis and interpretation, where the evaluation and interpretation of any piece of evidence depends on lots of prior knowledge from elsewhere - on what Bruno Latour calls Black Boxes. But this would add to his argument.)

What Dilbert is rejecting is the theory of repetition, whereby if you repeat something often enough it becomes true, or if you get enough bits of weak evidence from different sources, it becomes strong evidence. This is a theory that is embedded in the way that lots of organizations behave, and in the way that a lot of computer systems produce so-called "business intelligence". Dilbert is all-too-familiar with the ways in which false knowledge can emerge (or should I say evolve?) in complex social settings.

Technorati Tags: Darwin Dilbert knowledge science trust

Open Architecture

Philosophers from Kant to Peirce have discussed the structure of knowledge in terms of architecture.

Peter Lindberg (Tesugen) finds some interesting data to support this.

• Quotes from Kant's Critique of Pure Reason (first published 1781, via Amazon)
  
• José Ferrater Mora, Peirce's Conception of Architectonic and Related Views (first published 1955)

In these quotes, Kant identifies some characteristic functions of human reason and science:

• Human reason regards knowledge as belonging to a possible system.
• It therefore allows only such principles as do not at any rate make it impossible for any knowledge that we may attain to combine into a system with other knowledge.
• Science is architectonic, in that it makes a system out of a mere aggregate of knowledge

Kant presents our knowledge system as an open system - we are (or at least should be) always open to new knowledge that may trigger us to reconsider and reconfigure the knowledge we already have. This was a radical idea for Kant's time. Previous philosophers saw knowledge in terms of recovery (restoring what had been lost) and accumulation (adding to the sum of what was already known) - in other words, knowledge as a closed system.

Open-mindedness may be either an individual characteristic or a collective one. A person or an organization can be open or closed to new ideas and information, and can be tolerant or intolerant of uncertainty.

We can observe many settings where knowledge and uncertainty seem fragmented and brittle, and where they seem strongly resistant to reevaluation and change. In such settings, there is often a strong drive to add more knowledge, with more detail and precision, and yet the extra knowledge fails to have any useful effect.

Kant invites us to pay attention to the architecture (or architectonics) of knowledge - how knowledge is composed and decomposed. Equally, we must pay attention to the architecture of uncertainty - how uncertainty is composed and decomposed. This may often be the best route to opening up an organization to innovation and positive change.

Technorati Tags: architecture Kant knowledge open Peirce uncertainty