Guys, this is nothing but the Forer effect. In 1948, a psychologist called Bertram R. Forer gave a class of his students a test and at the end gave them a tailored writeup of their psychological profile, asking them to rate it on a scale of 0-5 (with 5 being totally accurate). The average score was 4.26. Only afterward did he reveal that he had ignored the test results and given each member of the class the exact same profile, made up of empty statements culled from horoscope readings.

Let’s examine some of the Colorgenics results. Usefully, the URL of a test result set looks like this:

http://www.goldinuniverse.com/showprofile.asp?id=1/7/3/0/2/4/5/6/&name=PT%20Barnum

The numbers correspond to the order you clicked the coloured boxes in, so you can swap them around and quickly see the results of lots of different profiles. If you do so, you’ll quickly see the same paragraphs recurring. For example, 1/7/3/0/2/4/5/6 and 1/4/3/0/2/7/5/6 and 6/4/3/0/2/7/5/1/ are very different sets of answers, but they all say:

Being impulsive and irritable, your desires and needs are paramount. You do things with insufficient thought – with little regard to the consequences that may follow. As a consequence of this attitude, you may be experiencing stress and conflict.

This is classic Forer effect. It feels personal because it claims to have perceived a character flaw — “impulsive and irritable” — that you would like to keep hidden. But who can say that they haven’t, at some point in the recent history, acted impulsively or been irritable? It’s actually a universal truth. “You do things with insufficient thought with little regard to the consequences that may follow” is just “impulsive” written out again, with the extra wordiness acting to stress the point and lend the text an air of academic certainty. “[Y]ou may be experiencing stress and conflict” is another universal truth for pretty much anyone in the world. Even Tom Hanks alone on that island in Castaway was experiencing stress.

1/4/3/0/2/7/5/6 and 6/2/3/0/4/7/5/1 both finish with the paragraph

You would like to be respected and valued for yourself and this can only be achieved from within a close and harmonious relationship.

Again, if you consider this in isolation, it’s simply universally true of everyone. Everyone wants to be respected, and everyone fears that they aren’t. Everyone would like a close and harmonious relationship. Taken in isolation, these are simply pat truisms, not the dazzling insights they are presented as. I wouldn’t mind so much if the website was presented as a bit of fun, but dressed in all this hippy bullshit and selling something that sounds awfully like a pyramid scheme really gets my blood boiling.

If you’d like to know more, I highly recommend the excellent Richard Dawkins interviews Derren Brown YouTube videos. They discuss the Forer effect extensively, as Brown has used it as the cornerstone of several of his tricks.

THE doctor who sparked the scare over the safety of the MMR vaccine for children changed and misreported results in his research, creating the appearance of a possible link with autism, a Sunday Times investigation has found.

However, our investigation, confirmed by evidence presented to the General Medical Council (GMC), reveals that: In most of the 12 cases, the children’s ailments as described in The Lancet were different from their hospital and GP records. Although the research paper claimed that problems came on within days of the jab, in only one case did medical records suggest this was true, and in many of the cases medical concerns had been raised before the children were vaccinated. Hospital pathologists, looking for inflammatory bowel disease, reported in the majority of cases that the gut was normal. This was then reviewed and the Lancet paper showed them as abnormal.

How serious was the impact this paper had? “Rates of inoculation fell from 92% to below 80%”, notes the Times. The BBC also shows us with this chilling graphic:

Let’s not mince words. If these allegations are true, then it means Andrew Wakefield made stuff up, and children suffered and  — in two cases — died as a result. He should be struck off and face criminal charges.

Our nation, in which 29% of teachers believe Creationism should be taught in schools, needs to take a long, hard look at the appalling understanding of science amongst the general populace, and how vulnerable this makes them to manipulative snake-oil salesmen. Or to put it another way, as the awesome MissPrism said on her blog whilst writing about the legal troubles faced by Dr Ben Goldacre, “babies are not epidemiology qualifications“.

Edit — Richie has pointed me to an excellent post by Ben Goldacre, “The Media MMR Hoax“, where he analyses in much more detail how the public outcry started, and who started and propagated it.

I loved the book, but despite my attempts to pickle it with alcohol the PhD wielding bit of my brain did twitch a number of times during the book. I’d prefer to read some of the more startling conclusions in peer-reviewed journal papers, I think, rather than a pop culture novel that cannot hope to present much of the raw data the conclusions are drawn from. After all, extraordinary conclusions require extraordinary proof, and the entire raison d’être of the book is to present startling conclusions.

Some parts of the work seemed very solid to me. The aforementioned Sumo wrestling investigation is an outstanding piece of investigation (you can read more about this here, although note the bottom of that article contains a partial rebuttal to the conclusions in Freakonomics from an expert on Sumo). However, one bit that didn’t sit right with me was the question posed by this scenario:

Consider the parents of an eight-year-old girl named, say, Molly. Her two best friends, Amy and Imani, each live nearby. Molly’s parents know that Amy’s parents keep a gun in their house, so they have forbidden Molly to play there. Instead, Molly spends a lot of time at Imani’s house, which has a swimming pool in the backyard. Molly’s parents feel good about having made such a smart choice to protect their daughter.

But according to the data, their choice isn’t smart at all. In a given year, there is one drowning of a child for every 11,000 residential pools in the United States. (In a country with 6 million pools, this means that roughly 550 children under the age of ten drown each year.) Meanwhile, there is 1 child killed by a gun for every 1 million-plus guns. (In a country with an estimated 200 million guns, this means that roughly 175 children each year die from guns.) The likelihood of death by pool (1 in 11,000) versus death by gun (1 in 1 million-plus) isn’t even close: Molly is roughly 100 times more likely to die in a swimming accident at Imani’s house than in gunplay at Amy’s.

My problem with this is that it seems to ignore the fact that if anyone has a pool, they probably only have one; whereas if anyone has a gun, they probably have more than one. Thus, by extrapolating from number-of-guns-in-country to households, an important multiplier — the average guns per household — has been overlooked. The real difference in the odds (it seems to me) must be much less than the “roughly 100 times more likely” quoted here. However, as I doubt the average number of guns owned in any given gun-owning household is surely less than 100, the conclusions of the book still hold — but by a narrower margin than this would suggest.

These commenters on the official Freakonomics blog have a different criticism of the same conclusions, namely:

The question in the book about which is more dangerous.. swimming pools or guns… is complete rubbish. Children are not given the same access to guns are they are to swimming pools! Are you saying that if children were allowed play near and with guns to the same level they play near and with swimming pool there would be less deaths from guns?

I’m not sure I agree with this. It seems to me the 1-in-1,000,000 statistic already takes into account the child’s (hopefully small) access to the guns. It does go to show, though, just how tricky this sort of analysis can be.

Admittedly I think such nit-picking at the statistics shouldn’t be taken as a criticism of the book itself. The fundamental premise of the book is to present, in a series of worked examples, the results of using economics tools such as regression analysis on real-world problems they would not normally be applied to, and interpreting the results in a purely scientific manner even when there are contentious moral issues involved (such as the question, did crime fall in the USA throughout the 1990s because of the legalisation of abortion following Roe vs Wade in 1973?).

This is not to say that the moral dimension is not important, of course; it’s just that Levitt and Dubner conciously exclude it from their analysis. This aspect reminds me a little of Derren Brown’s book Tricks of the Mind, which discusses things like cold reading techniques used by fraudulent mystics. Both books have a similar undercurrent of an appeal for more rationalism and logic in people’s worldviews.

So, anyway, yes. A very recommended read.