Monday, 9 January 2006

[deal architect:] The Borg explains its compliance budget survey

Read this great post from Vinnie Mirchandani on how Gartner markets surveys without even believing they're credible: deal architect : Gartner explains Compliance Budget Numbers.

Upon seeing compliance represented as 15% of IT budgets (see Gartner: Sarbanes-Oxley Compliance Hits 15 Percent of the 2006 IT Budget, French Caldwell , 9 December 2005 –subscription required :-( Vinnie did his research and wrote to Gartner to question how they came up with this quite extravagant number. And the answers were:

  1. This number comes from a survey
  2. Survey respondent played up the survey -in other words, if compliance is in your job title, you have a vested interest to give a high percentage to justify your budget (and salary) with a nice Gartner piece of research. Oh, and yes, two years ago there were not a lot of Compliance Officers (or similar titles) around…
  3. More gets flagged as “compliance project”: there’s fear around this issue, it’s the latest hype and so it’s easier to get funding for those projects. Budgets are still roughly the same, IT organisations still do the same stuff, the names under which they report projects is just changed depending on what the board wants to see.

The lessons for AR professionals are:


  • You should ALWAYS ask yourself what is the research methodology that is used in a research piece: is it a survey, a vendor poll (were all the vendors represented polled?), a DRE etc…

  • For surveys, always check the survey sample size and distribution. In this case, the sample was only 326 distributed as follows: 60 IT compliance managers, 51 CIO’s, 41 Finance manager, 23 Internal audit (IT), 20 Corporate compliance officer, 17 CFO’s, 17 Internal audit (finance), 17 IT audit manager, 7 CISO’s (not CISO’s) and 73 “others” (or 22% of the sample!)…
  • The geographic distribution was not mentioned, so one could assume those figures are valid for US of A only? (it’s the country of Enron after all).
  • The distribution by industry is not mentioned either, one could suppose that if financial sector are over-represented it could skew heavily the results (they have simply more regulations to comply with).
  • You should also question how is the sample selected: is it randomly selected from a well targeted population (check this post from Jason Corsello, a Yankee Group Analyst, sniping at an Aberdeen survey -thanks to Fred for the link) or is it a self-selecting like this one by Quocirca with El Reg?
  • Check also the taxonomy: as graphically illustrated above, it is important to understand how the categories are made and what is counted. Is IT security part of compliance???

Bottom line: understand how the research is carried out, always question the results.

See also:

3 comments:

Vinnie Mirchandani said...

I think you are being a little hard on French Caldwell at Gartner.

I think compliance costs are out of control in the US. In one of my posts on my blog I used the 15% of IT Budgets number from Gartner's survey. A couple of readers told me they thought it was too high. I emailed French to see if he could explain.

I thought it was awfully generous of him to write almost 2 pages to explain . He was under no obligation to do so - I am not a paying client.

I also appreciated his very honest answer. It shows us the games being played around compliance budgets.

My broader takeaway from his answer for your vendor clients is different than yours.

It is ok to be honest in your external communications. Every thing does not have to be massaged and packaged. In the growing world of blogs, honest, transparent behaviour will earn you far more points. Just eee what Robert Scobble is doing in his blogs at Microsoft. He generally takes the party line, but where he cannot defend it, he says so.

We need to encourage analysts like French to be similarly transparent. I certainly appreciated his taking the time to respond to me.

Duncan Chapple said...

"only" 326?? That's a massive number, and given your comments I can't believe that even a sample of 100% of CIOs would give you numbers you'd be happy with. In reality, the compliance manager has control of resources that pre-existed. This problem is more likely to be a question of sloppy questions: rather than only ask how much of the IT budget is concerned with compliance [which could be a large portion, since it involves core systems] one could have asked how much the IT budget needed to increase to meet the requrements of compliance.

French Caldwell said...

Good point -- for our next survey, I'll look at adding that question on how much the IT budget needs to increase to account for compliance.