Friday, 25 November 2005

79.7% of all Gartner RAS statistics are made up

Gartner RAS analysts are constantly spouting off very authoritative sounding statistics, but is there anything behind those pronouncements?

It’s important to distinguish between the analysts who generate market share number for the Dataquest services and those that primarily deal with end users, AKA IT buyers at corporations and government. The analysts supporting Dataquest services, go out and do the hard work of getting the numbers from vendors and doing their best to validate the information. Dataquest analysts might not always be right, but at least they are making the effort.

On the other hand, the traditional RAS (research advisory service) analysts that deal with end users do very little systematic research. The RAS analysts count on their informal conversations with a statistically small and invalid population of self-selecting clients for much of their information. Most of the rest of the information then comes from vendor briefings. In both cases, none of the data points gathered goes into a knowledge management system or a data warehouse for systematic analysis – it all resides between the ears of individual analysts. Long time Dataquest analysts when discussing their RAS colleagues say “For those RAS guys one data point is a trend, two is confirmation and why bother with three?”

After a few beers, Gartner RAS analysts will tell you that many of their statistics and numbers are DRE – direct rectal extraction. Should end users and vendors be making critical decisions based on recommendations pulled out of some analyst’s behind?

4 comments:

Anonymous said...

As an RAS ex-analyst I must admit you are right.
DRE often was used to validate intuitions.

Cannot say that all analysts do the same, but it is a usual habit.
However, as long as they are presented as "qualitative" results I think there is no unfairness.

Silicon Valley Guy said...

I agree that if the analysts characterized their pronouncements as opinion or qualitative, then there would be less concern. Unfortunately, this does not happen -- most analysts make these statements in such a way as to imply that they are based on hard facts.

Gartner use to have a device called a "stalking horse" that was not suppose to be based on hard facts, but preliminary speculation. Have not heard that phrase used much lately. Maybe Gartner got too much pushback from end users who thought they were paying for hard information.

Anonymous said...

79.7% are made up. ROTFL. You sure it's not 79.68% or even 80.31%? Do us all a favor and post your "research data" since you must have read (and understood) a LOT of RAS statistics. And you talk about DRE...

Silicon Valley Guy said...

Hi Anonymous,

My use of "79.7%" was meant to be a bit tongue-in-cheek, not to be taken seriously.