[deal architect:] The Borg explains its compliance budget survey

Read this great post from Vinnie Mirchandani on how Gartner markets surveys without even believing they're credible: deal architect : Gartner explains Compliance Budget Numbers.

Upon seeing compliance represented as 15% of IT budgets (see Gartner: Sarbanes-Oxley Compliance Hits 15 Percent of the 2006 IT Budget, French Caldwell , 9 December 2005 –subscription required :-( Vinnie did his research and wrote to Gartner to question how they came up with this quite extravagant number. And the answers were:

1. This number comes from a survey
   
2. Survey respondent played up the survey -in other words, if compliance is in your job title, you have a vested interest to give a high percentage to justify your budget (and salary) with a nice Gartner piece of research. Oh, and yes, two years ago there were not a lot of Compliance Officers (or similar titles) around…
   
3. More gets flagged as “compliance project”: there’s fear around this issue, it’s the latest hype and so it’s easier to get funding for those projects. Budgets are still roughly the same, IT organisations still do the same stuff, the names under which they report projects is just changed depending on what the board wants to see.

The lessons for AR professionals are:

• You should ALWAYS ask yourself what is the research methodology that is used in a research piece: is it a survey, a vendor poll (were all the vendors represented polled?), a DRE etc…


• For surveys, always check the survey sample size and distribution. In this case, the sample was only 326 distributed as follows: 60 IT compliance managers, 51 CIO’s, 41 Finance manager, 23 Internal audit (IT), 20 Corporate compliance officer, 17 CFO’s, 17 Internal audit (finance), 17 IT audit manager, 7 CISO’s (not CISO’s) and 73 “others” (or 22% of the sample!)…
  
• The geographic distribution was not mentioned, so one could assume those figures are valid for US of A only? (it’s the country of Enron after all).
  
• The distribution by industry is not mentioned either, one could suppose that if financial sector are over-represented it could skew heavily the results (they have simply more regulations to comply with).
  
• You should also question how is the sample selected: is it randomly selected from a well targeted population (check this post from Jason Corsello, a Yankee Group Analyst, sniping at an Aberdeen survey -thanks to Fred for the link) or is it a self-selecting like this one by Quocirca with El Reg?
  
• Check also the taxonomy: as graphically illustrated above, it is important to understand how the categories are made and what is counted. Is IT security part of compliance???

Bottom line: understand how the research is carried out, always question the results.

See also:

• AR 101: the research process
• More on Analysts’ Data Sample Size
• My customer is not your client, or why RAS analysts need to go beyond DRE
• 79.7% of all Gartner RAS statistics are made up
• The Gartner Magic Quadrant: shaken, not stirred