Forrester Gets It Wrong On Facebook (Really Wrong)

I’ve read the Forrester research on Facebook Facebook, and it makes very little sense.

Forrester has basically said that the entire value and promise of the Facebook brand is vastly overstated because of the lack of utility of mobile advertising, and they site their survey research as a demonstration of why this is, in fact, the case.

Forrester goes so far as to write in an open letter to Facebook CEO Mark Zuckerberg that “Facebook is failing marketers.”

On the surface, this seems like a truly outrageous statement, as Nate Elliott from Forrester even notes in his blog. After all, Facebook collected $5 billion in advertising revenues last year alone.

But if one looks deeper into the data and methods employed in this study, one can clearly see that the statement that “Facebook is failing marketers” isn’t only outrageous on the surface – it’s outrageous on all levels.

As a practicing survey researcher who has advised brands – not including Facebook – I well understand the limits of survey research. For Forrester to make the absolute determination that they have, based on a survey of 395 marketers and eBusiness executives, is simply wrong, inappropriate, and well beyond the scope of the type of research they’ve done.

Moreover, in ranking different media in the way that they did, the differences they report are sufficiently small that it is inaccurate to make the kind of definitive and absolute judgment that they did, based on a relatively small sample, and based on relatively small differences.

For instance, the difference between satisfaction with Twitter marketing and Facebook marketing is just 0.03 points (based on an overall scale of 1-5). This is hardly the kind of figure an analyst should use when making such broad, sweeping judgments about a company’s effectiveness in generating business value.

Indeed, the difference between the lowest and highest source of advertising was within the statistical margin of error – the entire spread was a mere 0.3 points on a scale of 1-5. This means that there is not a statistically valid difference between the highest rated source of advertising, on-site ratings and reviews, and the lowest rated, which was Facebook.

Further, when respondents were asked their level of satisfaction with six major online companies as marketing partners, Facebook fell in the middle of the list, ahead of Twitter and MSN, and virtually tied with Yahoo Yahoo. Forrester’s open letter to Facebook makes no mention of this finding, underscoring the bias, exaggerated position the letter takes. Yet the significance of these findings should not be overstated either, as the difference between the lowest and highest rated company was 0.56 points – once again well within the statistical margin of error.

Thus, I think that the entire Forrester survey is of no great utility and benefit, and is certainly not one that should allow anyone to make any significant or substantial judgment about the value of the Facebook brand.

To be sure, there are real questions about the validity and utility of mobile advertising.

In my experience, it is a question that is still very much up for determination. But for Forrester, based on a paucity of evidence, and data that is not statistically definitive, it is misguided at best to market their results as conclusive. In my judgment, Forrester’s interpretation of the findings is, to put it simply, wrong.

As we’ve seen this week with the gyrations in price, what is going to drive Facebook in the future are its earnings, whether good, bad, or indifferent. That is as it should be. But it does not serve the cause of intelligence and useful analysis, nor of logical discussion, for Forrester to make the kind of definitive and arguably extreme judgments they have made based on their research.

As I indicated, I have not worked for or had any contact with Facebook, but as someone with 40 years of working with brands and an equal number of years spent as a commentator on survey research and marketing, I know when I see surveys that have been used well beyond their value and purposes.

This is one of them, and Forrester would do better to use more reserved speech, and arguably limit the conclusions they draw from data of this type.