Could NPD be Wrong?

[famousringo]

BengaBenga said:
Well yesterday's NPD sure is confusing.

Pachter, EEDAR and VGC all had GTA: CW over 200k for example (Pachter even 450k). Now sure we can say that Pachter and Divinich are "mere analysts" but they DO research. I think GTA: CW under 80k is too much of a discrepancy tbh.

I don't live in the US, so I don't know how bad the situation is there, but a 33% collapse in hardware seems extreme too.

However NPD does have the highest data coverage, so should be statistically closest.

That last sentence isn't necessarily true. It's possible to have a large sample size, and yet that larger sample could be more biased than a smaller, more representative sample.

For example, suppose that three months ago, NPD had a model which accurately accounted for that Wal-Mart-sized hole in their data. Now suppose that the recession has caused customers to flock towards Wal-Mart and away from other retailers. These customers buy just as many video games as they would have at Target of Gamestop, but they're trying to spend less on other goods at Wal-Mart and buying their games while they're there. In this fairly plausible example, a video game tracker which samples 5% of the market, including Wal-Mart stores, would see no change in game sales. NPD, with 60% market coverage but no Wal-Mart coverage would see a contraction in video game sales.

I'm not saying that this is actually what's happening, and I'm not necessarily saying that VGC, EEDAR, and Pachter are more accurate than NPD, but size isn't the only thing that counts when it comes to samples. In fact, size doesn't count for much after you get 1000 samples. Making sure your samples actually represent the population and aren't heavily biased is very important.