With all of the new faces that are likely to show up in light of the recent NPD news, I thought I'd post a thread giving an example of how statistical sampling works. (I also want to have a post to link back to whenever someone insists that the numbers on this site are "made up.") We'll keep this pretty simple, leaving out things like confidence intervals and normal distributions, while focusing on one example.

The premise of statistics is that we can look at a small portion of some large group, and use it to make very accurate predictions about the group as a whole. While there will always be discrepencies with the "real" number in the larger group, we can get an extremely good idea of the overall picture so long as we have a true random sample. Let's look at an example.

Probably everyone here has heard of gamefaqs.com, which conducts a poll each and every day. I snapped a picture of one of their polls right at the start of one day:

This question on favorite puzzle series is a good one to use, because there shouldn't be any fanboy bias to deal with. (This helps us get a random sample.) Notice that there are only 139 responses so far. Think of these numbers as the estimates produced by VGChartz. We have a small random sample of a much, much larger total - basically the same relationship that VGChartz has with retailers and sales.

So how accurate of a picture did this particular small sample end up producing? Here's the same poll at the end of the day:

We now have over 67,000 votes on the same topic. Think of this as the "real" sales data VGChartz is trying to track. The initial sample tracked only 0.2% of the total - that's 1 out of 500! But surprise! The overall picture turns out to be extremely accurate. Our tiny sample correctly indicated that Tetris is by far the most popular game, with all the others trailing behind.

Now if you look closely, you'll see that there are some errors in the sample. Tetris is overestimated (72% to 64%), Bust a Move and Lemmings are undertracked, and Pokemon Puzzle League is noticeably too high. In fact, our sample incorrectly had Pokemon Puzzle League ranked higher than Bust and Move and Lemmings. This is exactly the sort of cherry picking that doubters use to "disprove" VGChartz. But to say this is to miss the forest for the trees; individual elements in a statistical sample can definitely be off, especially when the numbers are close together. Clearly, however, the numbers from our sample were not "made up"; even with our tiny sample, it nailed three games almost exactly (Columns, Mr. Driller, Puyo Pop) and was reasonably close on two more (Tetris, Pokemon Puzzle League). Most importantly, the overall shape of the group is extremely clear from our sample. The three tier structure (Tetris alone, followed by close numbers for Bust A Move/Lemmings/Pokemon Puzzle, and then a trailing group of the other three) of the group immediately jumps out from both graphs.

So while the sampling methods VGChartz uses will often be wrong on the micro level (due to margin of error/confidence interval reasons), it will very rarely be wrong on the macro level. For those who would continue to doubt, try looking at how other samples are put together. You'll find that it's very possible indeed to look at a couple hundred of responses and draw inferences about tens of thousands, or even millions, of pieces of data.

VGChartz may have an easier task than that ...

If you have a statistic which you can determine is the 'real' value (or at least an additional statistic you can judge your values against) you can start to figure out where the bias' in your sample come from and attempt to compensate for them.

One problem with this: vgchartz gets numbers from a few places every time, it's not randomly distributed (and different each time) like the first 139 people to vote in a gamefaqs poll. The first 139 people to vote in that poll are probably fairly representative of the 67 thousand who voted, as they are random.

For example, if gamefaqs.com polled 139 specific people every time they had a new poll, from the same 50 different towns throughout the US, that would not be as good for predicting how the entire 67 thousand would have voted. Of course this is true of not only vgchartz but also NPD.

The way to get the numbers more accurate is to increase the sample size (I believe NPD was 60% or so), and hopefully vgchartz is always trying to do this so the numbers get more and more accurate over time.

statistics 101

Well, i'm pretty sure Ioi likely uses a mathmatical formula based on past history and trends to estimate about what percentage of the market each store "speaks" for on average to get the numbers as well.

Which is more like... your 3rd or 4th class in Statistics... or maybe earlier depending on what kind of major your stats class is in and which statistcal program you use.

Which do you use anyway Ioi? You don't painfully do it by hand i'm assuming. I perfer SAS, but I mean i've only used SAS, SPSS and like... Excell.

Sullla, very informative, yet easy to read post.

@sinha, while what you say is mostly true, each store will not sell to the same people every time. It may do so some of the time, but not always. I'm not your typical EBGames kind of shopper, as i typically go to Walmart/target, but occasionally i do shop at EB, Best Buy, Circuit City, wherever if its more convenient.

The random sample in the first picture may not be as random as you think - time zones will exert a significant bias as different parts of the world become the most active on the internet. Allowing for this, your sample would be extremely accurate.

Nice post that puts the whole "Where do VGChartz numbers come from/VGChartz numbers are wrong" to rest. ThX!

I posted something similar yesterday and I feel that a sticky with the "where do VG Chartz numbers come from" that contains this information would be essential... I would add a little more detail on confidence intervals as it is prudent to point out that a smaller sample sizes generally provide a larger interval (margin of error) around the mean than larger samples do. It bears noting that to get a 100% confidence interval -- i.e. our "number" has a +- of ZERO, we are basically saying we have the EXACT number which requires a 1:1 sampling rate... Not possible in the real world and not cost effective, either... That helps illustrate why we use sampling/statistics in the first place... It also helps illustrate that NPD, Media Create, Famitsu, and others are also subject to error as the don't sample 100%, either...

Excellent post. Though you'd think more people would have this basic understanding...