New York Times: Gamers Agree With Critics

This spring a game-company analyst at UBS named Ben Schachter published a report with a subversively innocuous title: “Does Video Game Quality Impact Sales?”

...

To conduct apples-to-apples comparisons, I used the scores at Metacritic.com, which pulls together multiple reviews to give media products a score ranging from 1 to 100 that reflects the collective wisdom of professional critics.

The data was clear. The Top 10-selling games of last year — including titles like Gears of War and Guitar Hero 2 —had an average Metacritic score of 87.5. Only one of the top-selling games scored less than 80. (More about that later.) Meanwhile, the Top 10 box-office films of last year — including titles like “Pirates of the Caribbean: Dead Man’s Chest”’ and “X-Men: The Last Stand” — collected a poor average score of 62.9.

...

“When something costs $59 and you’re 18 years old, it’s not like going to the movies,” said Jeff Brown, vice president for corporate communications at Electronic Arts, the No. 1 game publisher. “Video-game consumers are the single most sophisticated shoppers in the entertainment industry, despite their age and what you might think. People go to a movie or buy a book on a whim. But buying a video game is a much more methodical and judicious process. Next time you’re in an airport, look at how many magazines are dedicated to video games. And they’re not writing about the sex lives of game designers. They are writing about the content.”

(from Kotaku and Game Set Watch)

--------------------------------------------------------------------------------------------------------------------------------------

Survey: Game Score-to-Sale Theory Disproven (2005):

The new issue of the Susquehanna Financial Group's Video Game Journal contains an in-depth analysis of the correlation, or lack thereof, between video game review scores and the resulting performance at retail. According to the authors of the study, Jason Kraft and Chris Kwak, there is little relationship between a game's critical and commercial reception.

Survey: Game Score-to-Sale Theory Again Disproven (2006):

In fact, the study found that by increasing the sample, "only 15.8% of the movement in game unit sales ... can be explained by movements in game ratings," a figure down from its original 17.3% from the sample of 275 games.

To help try and trap for other factors that might explain the findings, the authors looked specifically at franchise correlation, including sequels and new franchises, and selected data from other consoles, in all cases failing to find sufficient direct score to sales matches outside extreme isolated cases.

In conclusion, the authors state, "a theory (that game ratings matter) that fails under scrutiny is accepted as conventional wisdom. Conventional wisdom is wrong. And we have not even addressed the causation argument – that a higher rating causes a game to sell better. There is no reason to argue causation, because while correlation does not equal causation, the absence of correlation means no causation in our case."

--------------------------------------------------------------------------------------------------------------------------------------

FYI, I tried to look at Metacritic scores and game sales in a previous post.

I'm flabbergasted.

nice

Doesn't surprise me that it's hard to nail down a real correlation. Review scores are going to be biased towards games preferred by people who frequent review sites, lowering the scores of games like The Sims and Wii Sports. Advertising can certainly have a tremendous effect, as can the release of other games at around the same time.

I'd say that the topmost study is rather flawed in that it only looks for one-way correlation - it checks to see if the top-selling games also had good reviews. Let's ignore for the moment the fact that an average of 87.5 doesn't really imply consistently high ratings in this industry (where an 80 is a 'pass' for most gamers). Plenty of highly rated games just didn't sell that well. Look at Okami or Psychonauts.

Of course the games that sell the best are highly rated. They're almost always incredibly well-hyped, being either flagship games like Gears of War to which the hopes of a system are pinned or returning blockbusters like Halo and Zelda. As such, they enjoy massive budgets and the best talent in the industry, and they benefit from whatever pressure there is for reviewers to be positive about games that they know will sell millions.

The best games are going to be highly rated, but everything else is going to be a wash. Far more important than quality are timing, advertising, and franchise.

If you've ever been curious as to why Majesco is all about the Wii ...

They produced one of the most critically acclaimed games of the previous generation (Psychonauts) and even with the more modest development budgets of the previous generation its poor sales almost bankrupted them.

The fact of the matter is that the best reviewed games are typically designed to apeal to the very small core market which buys a lot of games; the unfortunate thing is if too many 'good' core games are released at the same time there are not enough consumers to ensure good sales for all of the games.

Very interesting.

that gave me a head ach to read

I don't take issue with the would-be debunkers' study design, but I DO disagree with their interpretation of the results. If 15.8% of the variability in game sales can be explained by variability in game ratings, that means the correlation between sales and ratings is r = .397. A correlation of this size lies somewhere between a medium-sized effect (one noticeable by a careful observer, about r = .30) and a large effect (one obvious to an observer, about r = .50; Cohen, 1988, 1992; http://en.wikipedia.org/wiki/Effect_size).

The bottom line: 15.8% might sound like a small amount of explained variability, but most statisticians would consider this evidence of a moderate-to-strong association between game ratings and game sales!

I wouldn't call them "would-be debunkers", it was just an earlier study on the same subject (the study in the New York Times is the most recent one), and that's the only reason I included it, as relevant background information.

Sorry if that was unclear, I just did a quick copy-and-paste job instead of really explaining things.

Personally I think both GochayeX's criticisms of the new study and cjsoto's criticisms of the old study may have some validity. Actually, I really have no idea, but I still find this stuff fascinating!

There are some issues here. One of them is that the people who rate games are the people who buy games, and that's an effect of having a young industry. For the most part, there are no generational differences, few fleshed-out genres, and a very bland pallet of tastes overall. To say that gamers are more sophisticated than moviegoers is ridiculous. We're just a far more inbred group, and share a lot of common tastes. Once you get more diverse games, more diverse players, and get to the level where newspapers have video game reviews right next to their movie reviews, and the video game guy doesn't look like the movie guy's nephew, *then* statements like "Video-game consumers are the single most sophisticated shoppers in the entertainment industry" could be made.

But they'd probably be wrong.