Friday, 23 December 2005

AR 101 series: the research process

In this post, the ubiquitous James Governor points to a post by a Richard Monson-Haefel, an analyst at Burton Group (a US RAS client-side firm): Burton Group’s Brutal, but Effective Review Process. It is an interesting and detailed insight into a firm's research process.

ARmadgeddon's tips: AR professionals should do their research and me aware of the research process followed by the firms they track. Questions they should be able to answer are:

  • Do you have a research agenda?
  • Do you publish different kind of notes?
  • If yes, do they follow the same validation process?
  • Do you review research notes internally?
  • Do you systematically send your reports for vendor review?
  • Even if the vendor is not a client?
  • Do you send the whole document or just the portion related to the vendor?
  • Do you accept discussion on your positions and recommendations?
  • Do you communicate on changes made?
  • Do you take calls to discuss the review?
  • Even if the vendor is not a client?
  • What turnaround time do you expect from a vendor?

Tuesday, 20 December 2005

An after thought on Serendipp

Lunchtime are made for Google, aren't they? I've been trying to find out a bit more about our new friends at Serendipp, Ahmed and Dominic. Dominic used to be COO at SalePlane, so I ran this Google search to find out more. For some reason, the one link that comes up goes through to a Metropolitan Police warning about internet fraud.

Monday, 19 December 2005

Forrester's 2005 newfies

Just so that it's convenient for everyone, here are Forrester's 2H05 new EMEA hires:


  • Larry Velez, Global and European WAN procurement; VoIP communication architecture, infrastructure development (ex. META)
  • Jenny Lau, Advice to mobile operators on strategic issues like mobile commerce, 3G mobile networks and services, and enterprise mobility; strategic advice to mobile telecom vendors and firms looking to launch mobile services
  • Pascal Matzke, European IT services and outsourcing trends, as well as underlying vendor market dynamics; IT services and sourcing strategies (ex. META)
  • Euan Davis, European IT services and outsourcing trends, as well as underlying vendor market & end-user dynamics
  • Peter O’Neill, Focus: Infrastructure Management (ex. META)
  • Name can’t be disclosed – Security Analyst
  • Cliff Condon, How European financial institutions can use emerging technologies to improve the retail customer experience (channel distribution, self-service delivery, changes in business models)
  • Luca Paderni, Consumer marketing, consumer trends, branding, consumer strategy, online advertising, and online retail (including online banking)
More on Forrester in Weekly roundup: Forrester, IT-director and Quocirca (Part II)

Saturday, 17 December 2005

Ahmed Argues Against ARmadgeddon

Ahmed Siddiqui has written in self-defense against our earlier post:

As someone who has worked as an analyst at senior level (and who knows many other senior level analysts intimately), I reiterate and clarify: the source of most analyst opinion and data is from market sizing and developing case studies of clients, vendors and markets. How this is actually used is secondary to your own original point. It can go into published work or can form the basis of the credibility and trust used to win consulting work. Any analyst who cannot understand the market sizing process or create case studies- whether it be for a report, building an RFP for a client, or advising on vendor capabilities- isn't worth his/her salt. Market sizing and developing case studies is pretty much must of what analysts do. And as my posts says, the AR function is still important for this, just as it is for the bigticket consulting work.

I would love to hear readers' comments on Ahmed's opinion. Personally, I am baffled by it. Ahmed worked for Datamonitor's London headquarters, and his account makes me think that most of his time was spent sizing markets and writing case studies. I understand that as individuals, we primarily judge the world on our own experience of it. However, it seems to me to be self-evident that most analysts outside Datamonitor are not spending most of their time sizing markets and writing case studies. Market sizing is an activity primarily conducted by vendor-facing analyst firms, or by specialist teams like Dataquest. Many analysts are looking at the features and benefits of solutions, rather than the market shares of vendors.

Ahmed knows many analysts, as he points out. However, his fresh face compares favorably to my wrinkles. I deeply believe that I have met, and worked, with many, many more over the years [and years]. Very few are focussed on market sizing, outside IDC and Dataquest. As for case studies: how many analysts send more than a day a month working on them?

The gap in our viewpoints astonishes me. I appreciate that analysts are more quantitative in Europe, but are the differences so deep that mosts analysts there are engaged in such highly structured, and easily offshored, work? Can anyone think of a good reason why our views are so distant?

Friday, 16 December 2005

Do analysts only size markets & do case studies?!?

The wonders of Trackback! I've discovered Serendipp's sniping at us and Duncan. We are "almost hedghoglike" while Duncan is "bilious".

I am not familiar with Serendipp, but it seems they are AR advisors in England. Good luck to them because, if their blog is evidence, they have a lot to learn.

They feel that we and Duncan are missing the point in our discussion about analysts and how deeply they need to research: 'Analysts only do two things: size markets and do case studies'.

This is so far from being right, it's not even worth calling wrong. Sizing markets and writing case studies is what gets outsourced to India. The real work of analysts is quite different: and that is why analysts have substantial advisory, consulting and custom research businesses. If an AR advisor doesn't realize this, then heaven help their clients.

It would be far better to understand one's limitations and keep only do what you know how to do. In that respect, kudos to Hill & Knowlton. If Duncan's latest post is correct (my Norweigan is rusty) their European AR is led by their partner agency in Oslo, Gambit, which hired a former IBM AR manager in 1999. It takes guts to admit when you need help, but it's much wiser than pretending you know what you don't.

Feedblitz now available

Thanks to James' tip, we've set up Feedblitz to allow readers to receive ARmadgeddon by email. Just enter your email in the box on the bottom left-hand side of this screen and bingo!

Or just follow this link:

Tuesday, 13 December 2005

C'mon guys, sell to us!

Analyst Relations is a quite stressing job as AR professionals sit between external and internal constituencies and try to make both happy. What AR managers don't need is more aggravation when they want to work with analyst firms, but it seems that it's exactly what's happening: analyst firms in general do a terrible job at selling themselves to vendors. One of the reasons is that analyst firms sales reps try to SELL when often analysts often feel their ethical boundaries are being pushed. We imagine the conversations inside some firms between sales and analysts sales going like: "Would you be nice to my client so that they can buy reprints?" followed by "Do you mean I should tell something I don't believe in?"

We'll leave them to decide where they want to draw the line, for instance and to their credit, the Borg don't sell white papers (but do read Post-scriptum on "Yankee, analyst impartiality and circular references" for the caveats).

However, we thought analyst reps could do with a few ARmadgeddon tips to improve their sales process. Most of them are very basic but we've seen those mistakes many times.
So, hear, hear, sales reps:

  1. Sales lead: track those those converations your analysts are having with vendors, invite yourself to brefings. Too often, reps have little idea of research agendas and of vendor marketing plans.

  2. Qualified prospect: follow-up with your AR person to see if there's a real need.

  3. Need identification: for complex projects, do schedule a conversation between the analyst, AR, the vendor's decision maker(s) and don't forget to include the consultants who'll end up doing the work. For speaking enagement, come back with topic and spokesperson real quick.

  4. Proposal (business): it's amazing how long it takes for IT Analysis fims to send vendors a proposal. Get it out, even if it's sketchy or you'll loose the sale. If it's a speaking engagement, you need to turn around in hours, not in weeks! An email will do, but please do us a favour: make sure you include VAT number, physical address, firm name and delivery dates…

  5. Closing: again a basic skill but it seems that some analyst firms can't be asked to call to close a sale! Amazing…

  6. Deal transaction: we've seen so many firms being rubbish at admin. Send the invoice and get paid!

Another common mistake is not fitting into vendors budget processes. If a vendor closes its fiscal in March, make sure you get in their FY+1 plans in before February!

Finally, don't make the common but fatal mistake of not involving AR: they can and will block your proposal if they feel it's not addressing a need or simply if you bypass them. Help them and they'll help you.

Monday, 12 December 2005

More on Analysts’ Data Sample Size

In “My customer is not your client, or why RAS analysts need to go beyond DRE” ARonaut makes a number of excellent points about the sample size and skewing of data points. Here are a few more issues to consider when using analyst commentary based primarily on conversations with IT managers:

Small sample size on any one topic – In its 2004 annual report, Gartner claimed that it receives 230,000 client inquiries per year. It sounds like a large number, but how statistically valid is it? The average number of end-user data points is probably less than 400 to rarely more than 700 per analyst per year or about eight to 15 per week. Remember that an analyst answers questions on a variety of issues. So, of those eight to 15 inquiries per week, how many inquires concern a specific vendor or its products? A couple, maybe a little more or a little less? Forrester has even a smaller sample size because its business model and its client base is smaller.

Limited to firm’s clients – Another issue is that data points are limited to clients of the analyst firms. Gartner’s CEO has stated that Gartner has only 15% of the possible end user market at companies. Does Gartner’s or Forrester’s client base represent a statistically valid sample of the overall IT buyer market?

Further limited to advisory seat holders – Another limitation is that only advisory seat holders can conduct inquiries with analysts. Thus, there might be hundreds of IT managers at a company but only a handful actually talk with the analysts using inquiry. Are the analysts hearing the full story from only a subset of the IT managers?

Limited still further by caller self selection – The advisory analysts passively sit and wait for clients to call them. Thus, their data points come from self-selecting callers. What motivates these callers? Probably the most common reason is that they are working on a project (e.g., buying an ERP system). The second most common reason to call an advisory analyst is likely they are having a poor experience with a vendor and want to check with the analyst to see if other companies are having the same experience and what the end user should do to correct the situation. It is unlikely that IT managers who are happy with their vendors are calling up the analysts simply to complement their vendors. As a consequence, analysts are only hearing from a vendor’s disgruntled customers, which is not a statistically valid sample.

Limited once more because of no follow up by analysts – Advisory analysts rarely proactively follow up with end users to see how something was resolved by vendors. Maybe within days or weeks the vendor corrected the situation to the satisfaction of the customer, but the analysts never hear about that. Thus, the analysts are left with only negative data points.

No data warehouse, knowledge management or business intelligence tools – None of the advisory analyst firms have invested in infrastructure to capture, store and analyze the information that surfaces during inquiries. As a consequence, the advisory analysts rely on their memory to store this information and we all know how faulty memory can be. In addition, this means that when an advisory analyst leaves a firm, all those data points leave with them.

Second hand information – The advisory analysts claim that the number of data points they work with is actually much larger because they pool data from the entire team. However, without a knowledge management infrastructure then this pooling is little more than sharing stories around a camp fire. Over time significant errors can creep into the share pool of information as second hand information is passed on from analyst to analyst.

Bottom line: The primary information that advisory analysts have is not statistically valid because of the limitations mentioned above and in ARonaut’s post. For instance, a vendor might have thousands, hundreds of thousands or even millions of satisfied customers, but the analysts are relying on information from only a few dozen or less disgruntled customers. While the inquiry-based information can provide interesting insights it cannot take the place of fact-based research from surveys and other systematic research.

Recommendation: All clients, end user and vendor, of the advisory analysts should take responsibility for checking the validity of analysts’ data from client inquiry. Question that could be asked include:

1. Pin the advisory analyst down on how many data points they specifically have on the relevant topic. Don’t let the analysts be vague or talk about how many inquiries that do they – this could be masking a very small set of relevant data points.

2. Pin the advisory analysts down on how many different companies are represented in the sample size. There could be a number of inquiries the analyst that the analyst could point to, but the number of unique companies could be much smaller because a few end users make multiple calls.

3. Have the analysts discuss the time frame that the data points were gathered over. There could be a problem if the data points are out-of-date or too recent.

4. Have the analysts discuss the characteristics of the companies that were the source of the inquiries. Are the companies similar to your company (if you’re an end user) or your target market (if you are a vendor)?

5. How much follow up has the analysts done to determine how effectively the vendor responded to the situation.

What other questions or issues should be raised by advisory analysts’ clients?

Sunday, 11 December 2005

Deal Architect on changing the influence game

Vinnie Mirchandani posted in his blog Deal Architect on The Changing Influence Game after discussing this with HP's AR chief Carter Lusher on this ARmadgeddon post (check the comments).

His views are worth a read: he suggests technology vendors need to look beyond the PR+AR+IR influence trio and start addressing procurement consultants (including Gartner's) and blogs. The comparison he uses in lieu of conclusion is striking -and smart.

While we agree that the sphere of influence is changing, we would also add that vendors need to do a better job at focussing on AR and ressourcing it properly. James give a couple good reasons why in this post: Analyst fingerprints on billions of HP’s revenues, Gartner 0wns GAO. Once it's covered, it is surely a good idea to address those second tier influencers... maybe with the exception of systems integrators (Vinnie mentions Accenture and we can't agree more that their consultants also need to be brought up to speed by tech vendors).

Friday, 9 December 2005

Weekly roundup: Forrester, IT-director and Quocirca (Part III)

This is the third instalment of last week's round up ‑following Weekly roundup: Forrester, IT-director and Quocirca (Part I) and Weekly roundup: Forrester, IT-director and Quocirca (Part II).

We'll conclude this serial post with Quocirca who are now actively recruiting analysts (wannabe must send CV to Clive) after loosing John Collins who went writing another music book (after Rush and Marillion) and Dale Vile who went with his wife Helen (also ex. Quocirca) to form Freeform Dynamics. They just formed a partnership with The Register to use El Reg as an outlet for their research and plan to tap onto its reader's base for research and surveys. It's interesting to note that Quocirca was (or still is???) a partner of The Register… Anyway, this seems like an obvious duplication with what Justin's boyz have launched on Wednesday. While we're talking about Quocirca and in guise of conclusion, Dave pointed out that Clive Longbottom has started a blog hosted on IT-Analysis (for more on this, see Weekly roundup: Forrester, IT-director and Quocirca (Part I)). But COME ON Dale, you need to post something now –perhaps not a novel every time tough! And also, we'll argue that this is not a real blog: as Catherine says here, you miss comments, trackbacks, serial entries and RSS. Tsss, no comments Dale, bad boy!

Thursday, 8 December 2005

Weekly roundup: Forrester, IT-director and Quocirca (Part II)

Following up on our previous post Weekly roundup: Forrester, IT-director and Quocirca (Part I), here's the rest of last week's round-up.

On Friday, Forrester organised a breakfast for AR professionals, which was very well attended with over 50 AR (lost of blue chip AR but also some AR agencies and smaller firms represented). On the Forrester side, we had Ian Taylor, Principal for the Oral Programme (their AR "club" with membership fee), ex. META Group Consulting Peter O'Neill, now turning infrastructure analyst, Martha Bennett, VP & research director for their European financial practice and Luiz Olivera, runs the ARM council in Europe. No execs, just the ubiquitous Dunc' who pitched on AR rules: be candid was the key takeaway.

The rest content was... interesting. Martha came out very well with content and with her top vendor peeves about analysts and vice versa (Read David's post for the detail: Forrester: Biggest moans about vendors). All entertaining. They also provided some trends on the industry, and amongst other things predicted IT vendors would stop selling directly to users by 2010. Things got even more interesting when some vendors asked if they intended to cover hardware -which in Forrester's view is becoming less strategic/important. Peter O'Neil was at pains to explain why he got the job to cover infrastructure, when his new colleagues predict it's going down the drain. Someone asked a quick run down on Forrester's strategy and they seemed to collectively struggle to answer. Martha hopefully gave an elevator pitch but they were clearly not at ease. And Luiz rushed through the Analyst Relations and & Marketing council, which is a combination between a gated community and a RAS seat. At £21k a pop, including event tickets and a dedicated contact. Oops.

The opinion of the AR professionals we spoke to was that the event was inconclusive at best, we learnt nothing on Forrester's strategy and differentiators, nor did they give vendors any good reasons why they should do business with them.

So what was the point? For AR professionals, great networking opportunities as David puts it. On Forrester's side, it's quite unclear what they were trying to achieve: maybe this was a lead generation for the ARM council but they never really got down to the sales pitch. It's quite striking that Forrester, alike most of its competitors, don't know how to address the AR community –one of its key decision makers/influencers! The pastries were nice though, thanks for the brekkie...

Wednesday, 7 December 2005

Weekly roundup: Forrester, IT-director and Quocirca (Part I)

This is in fact the roundup for last week, which has been quite eventful indeed.

On Wednesday and
at last, Justin Speake and his mates re-launched the newly spun off IT-Analysis, as a multi-analyst firms delivery vehicle. In other words, it’s a web site where research from various firms such as Bloor Research, Evalubase, Hewson Group, Hurwitz, IE4C, Luenendonk, Macehiter Ward Dutton, Metra Martech, Quocirca, Sageza and TechConsult is hosted. Delivery is free for most but the web site sells research through Infoedge, guidance through Amazon and is also supported by advertising (has anyone got memory problems?)

For more details, both David and Duncan (interesting to note that he was invited to speak but never made it to the stage –let's blame Lewis PR for that?) have published extensive posts on this:

We'd agree with David that everyone wins: small analysts get more exposure (fragmentation of the independent analyst world makes it difficult to navigate), users get a lot of research in one place and vendors get more visibility for sponsored research.

Overall, the business model is interesting and probably over time an alternative to the Borg: is that a first step towards
open source analysts? We'd love to hear James' opinion and even more to see him join!

Just one
comment for Justin though: you should replace the term "articles" by "research notes" or "reports"…

(to be continued)

Read also our previous post on Justin's plan:
Have you seen my plan, my plan, my plan?

Tuesday, 6 December 2005

Microsoft boosts its AR

Gartner Watch reports that two ex. Borg Members have joined Microsoft: Anne Marie Roussel, Mika Kramer. Are they going in AR or is it a sign that Microsoft is ?
In addition, we heard they've hired Naomi Higgins from Peoplesoft.

Monday, 5 December 2005

ARmadgeddon keywords

This month brought an interesting set of keywords leading to our blog:

  • armageddon ovum
  • kgc association
  • microsoft pr
  • west kensington
  • fred amoroso

Interesting indeed...

And as far as links, quite a few interesting ones too:

Sunday, 4 December 2005

Advisory analysts do impact vendor sales and make/break products

Vinnie Mirchandani in a comment to Who do the advisory analysts think are their clients? wrote “…And let's face it... [IT buyers] make or break products and vendors - the analysts just report it. …”

This statement is patently ridiculous.

If the analysts were merely reporters, would the IT buyers spend over US $1 billion per year with the advisory analysts? No. Reporting can be gotten for free over the Internet.

Advisory analysts like Gartner RAS are not just reporters, but are extremely influential when it comes to specific sales deals and the success of a product. There have been a number of formal studies by the Analyst Strategy Group, Hill & Knowlton’s Penn, Schoen & Berland unit, InformationWeek, Optimize and SageCircle of how the IT buyers use industry analysts. In all these studies, the IT buyers themselves tell the researchers how extensively they rely on IT analysts for critical decisions like which vendors to put on a short list. The buyers use the analysts for this type of product decision making support for similar reasons why companies outsource payroll handling to ADP or their employee cafeteria to Marriott.

Besides these objective third-party studies, every vendor relate can specific instances where an analyst have negatively or positively impacted a sales opportunity. These sales impacts occurs because the analysts do more than mere reporting, rather they are active advisors to buyers and key influencers in the industry.

Saturday, 3 December 2005

Norma becomes president of rs-unix

It's a loss for the analyst relations world: today I heard that Norma Larosa has been hired as President of rs-unix, an IBM reseller up in San Francisco.

Norma will be well known to AR veterans as co-founder, with Efrem, of Kensington Group, Inc. (KGI).

Norma was been one of the most widely respected and trusted AR advisors through the 1990s. KGI became part of MetaSource in 2002, but the new owners were not able to give KGI the backing it needed to grow. After KGI decellerated, it wasn't able to offer clients the sort of support they wanted. Of course, analyst relations needs are developing all the time and weak support from MetaSource wasn't enough to keep KGI ahead of the market.

Efrem returned to academia in 2002 (through he still does some research and training through Lighthouse). Through 2003 and 2004 Norma worked hard to keep some of KGI's services running for existing clients. KGI's website hadn't been worked on since 2003, and that reflects how she was focussed on meeting existing clients' needs. Earlier this year the KGI website was taken offline, and not much as been heard from Norma since.

In the long run, even someone with Norma's intense client loyalty cannot have her interest held by a low-growth mature business.Her skills are very broad, and rs-unix look set to gain.

KGI's collapse, like SageCircle's, shows the rapid maturing of the AR market. Nimble firms like KCG, ASG and Lighthouse have been able to deliver similar value at lower prices by copying those pioneers. Norma has not left a minute too early. We wish her every success in the future.

Friday, 2 December 2005

Who do the advisory analysts think are their clients?

Listening to advisory analysts give speeches at conferences or talking to them during inquiries or briefings, analysts say the word "client" all the time. However, who is this client?

The word "client" to an advisory analyst like Gartner or Forrester is almost exclusively meant to be end users, AKA IT managers at mid-sized and above organizations. Even though the vendors spend hundreds of millions with the advisory analysts, the vendor is clearly never thought of as a client, just a checkbook.

Thursday, 1 December 2005

My customer is not your client, or why RAS analysts need to go beyond DRE

Following up on 79.7% of all Gartner RAS statistics are made up, part of the mis-understanding between RAS analysts and vendors AR is down to the customer set surveyed, aka sample size and dispersion. RAS analysts (Gartner, Ovum, Forrester) need to recognise that they have often a much smaller sample size (amazingly, not everyone consults the Borg!) and this sample may be distributed quite unlike a vendors client set. Another factor is that RAS analysts tend to (re)act to client queries. End-user clients tend to pick-up the phone to the consult analysts when thy have an issue, not when their application or piece of kit works just fine (happens apparently). This explains why analysts are often the bearer of bad news and are reluctant to cover stuff that does what it says on the tin. For analysts, one could also argue that such behaviour drives their consulting revenue?

Another sample skew is about geographical and vertical distribution: vendors have a far greater sample size but obviously the dispersion is skewed towards specific industries or geographies where that vendor is stronger. So if you're doing great in say AsiaPac where analysts tend to be thinner on the ground, it's tough to get that recognised by analysts.

Finally, as as SVG points out, it would be nice RAS analysts could be a little more thorough with research and avoid discovering "trends" which are often only based on very few data points.

ARmadgeddon recommendation:

  • VENDORS need to be open with analysts, share installed base demographics and maintenance calls data (at a high level).
  • RAS ANALYSTS need to stop annoying AR with "my client this, my client that" statements and work on their own demographics too; they need to also start covering products that are not generating enquiries.