Wednesday, 8 March 2006

Grading pundits and prognosticators: More famous = less accurate

Here is an interesting column from Fortune Magazine about a University of California professor’s seven-year project to evaluate the accuracy of expert commentators Grading pundits and prognosticators: More famous = less accurate. While not directly on the IT analysts, it certainly applies to them as well. Here is an extract:

"… Another part of the answer is especially troubling for the media. The awfulness of Tetlock's experts was almost uniform... He found but one consistent differentiator: fame. The more famous the experts, the worse they performed. And of course it's those of us in TV, radio, newspapers, magazines, and on the web who bestow that fame. We're now reaching a deeper answer to the question of why experts persist. The media like experts, the more confident the better, though Tetlock found that more confident means less reliable. The media like them because you, the consumers of media, like them. Experts like to appear in the media because it pays. Tetlock's conclusion: "The three principals--authoritative-sounding experts, the ratings-conscious media, and the attentive public--may thus be locked in a symbiotic triangle. ..."

Too bad the IT industry doesn't have someone like Professor Tetlock to take a look at the analysts, especially Forrester, Gartner and IDC with all their predictions. The results might be eye opening.

2 comments:

Jon Collins said...

This is very good - and a lesson to us all.

Duncan Chapple said...

Actually some work has been done by some US academics. I mention them in this post...
http://analystrelations.blogspot.com/2006/02/no-alternative-to-megamistakes.html