In another thread (http://www.vgchartz.com/forum/thread.php?id=68497) Brett (aka ioi) said in response to questions about this month's NPD data:

Not as confusing as I do!

As I've explained many times - we get data, we extrapolate it, check it with other data then post it up on the site. I expect some differences with other tracking services - it's inevitable - but month after month I'm shocked with how the data NPD releases makes no sense and seems to yo-yo. Some months it is way higher than ours, other months way lower, and for no logical explanation!

Sure, no Easter this March, but that maybe has a 5-10% boost at best for the month. And with Halo Wars, Killzone 2, RE5, Pokemon Platinum etc hardware sales just nosedived? We see drops in our data Feb - March, as is normal, but not drops this big.

Everyone out there in the video game industry seems to take NPD numbers as the "gospel truth."  But the tracking agency seems to be more secreative than than the CIA (certainly, its more secreative than the US military).   And each month there seems to be prima facie questions about the data -- though no one seems to/wants to raise them.

We know that NPD gets data from Amazon.com but does not get data from Wal-Mart or Toys R Us. Could this data mix be toxic (providing incorrect results)? Could algorithsm from previous generations (when Wal-Mart and Toys R Us data was available and the console mix was different) be causing problems?

And why does no one ever query NPD for an explanation? The US seems to get the least public information about video game sales in the world from its "official" tracking service.  And it is less than we see for other forms of entertainment  (TV, Books, Movies, Music all have weekly rankings and some have mulitple sources and/or numbers attached).  But still everyone believes NPD and not any other source.

An alternative explanation is the "official' line that NPD knows what it is doing and others are just plain wrong. There are many who accept this as fact. But lately the NPD numbers have been too erratic (not just in comparison to NPD but with expectations and with past trends) to be accepted without at least asking some questions.

These are my thoughts.

Mike from Morgantown

It's actually pretty easy to prove that NPD's public data isn't completely trustworthy.

They never publish corrections to their past data. This means that either:

1- Their data is always correct the first time (quite unlikely given they don't receive sales data from Walmart and other big retailers)
2- They simply forget about past errors and carry on (adding up several months will result in wrong data, as well as each month's data being sometimes wrong).
3- They adjust for past errors by adjusting later months' data. In this case it's not possible to trust each month's data, since it may be wrong or adjusted up or down to account for past mistakes. At least in this case the sum of several months should be more close to the truth.

Take your pick, since we don't really know what they do. There's no transparency at all with NPD's numbers.

NPD might be a little off, but its the best the world has, and if their numbers are different from ours, we should adjust ours to fits theirs, seeing as they are more likely to be right.

Well yesterday's NPD sure is confusing.

Pachter, EEDAR and VGC all had GTA: CW over 200k for example (Pachter even 450k). Now sure we can say that Pachter and Divinich are "mere analysts" but they DO research. I think GTA: CW under 80k is too much of a discrepancy tbh.

I don't live in the US, so I don't know how bad the situation is there, but a 33% collapse in hardware seems extreme too.

However NPD does have the highest data coverage, so should be statistically closest.

Given some of the responses, I guess that's why I am asking the question.


@im_sneaky -- Is NPD the best the world has to offer? No one knows their true model or coverage? An error in the model can mean bad numbers.

@Bengabenga -- NPD has a lot of coverage. I wonder though if it is the right coverage, given who it is missing and the dynamics of the market, particularly this generation. If you are depending upon people going to Gamestop and suddenly they start going to Wal-Mart, there is no way for NPD to know about that. I don't know if that is happening -- but we know that NPD does not know it either.

Mike from Morgantown

couldnt see a complete colapse from being in storees the last month, but my region was not hit as bad as most. on top of that I have been sayin dont trust the NPD since way back last year at this time, they have had legal issues with manipulating numbers in the past

Even if NPD were as reliable as Famitsu and Media Create they could easily be 10% out... but they have a bgger market to track for one thing, so they could be 15% out on anything even at their best.

VGC has the potential to be much further off of course.

Do you have more info on the NPD manipulation?

I don't recall hearing about it.

Thanks

Mike from Morgantown

That last sentence isn't necessarily true. It's possible to have a large sample size, and yet that larger sample could be more biased than a smaller, more representative sample.

For example, suppose that three months ago, NPD had a model which accurately accounted for that Wal-Mart-sized hole in their data. Now suppose that the recession has caused customers to flock towards Wal-Mart and away from other retailers. These customers buy just as many video games as they would have at Target of Gamestop, but they're trying to spend less on other goods at Wal-Mart and buying their games while they're there. In this fairly plausible example, a video game tracker which samples 5% of the market, including Wal-Mart stores, would see no change in game sales. NPD, with 60% market coverage but no Wal-Mart coverage would see a contraction in video game sales.

I'm not saying that this is actually what's happening, and I'm not necessarily saying that VGC, EEDAR, and Pachter are more accurate than NPD, but size isn't the only thing that counts when it comes to samples. In fact, size doesn't count for much after you get 1000 samples. Making sure your samples actually represent the population and aren't heavily biased is very important.

The only problem with ioi's Killzone2 mention is that it falls with Sony's own announcement that the game just passed a million... and not 1.4 million stated on VGC.