How accurate is VGChartz? - 12 week analysis

[Killergran]

We all know that VGChartz is being attacked on various fronts for not having accurate data, and for the numbers to be subject to change at any moment without notice. To gain some insight into the validity of this critisism and satisfy my personal curiosity I started The Weekly adjustments thread. I've been using a spreadsheet to do the simplest calculations I could possibly do on VGChartz hardware adjustments for the three consoles in the three big terretories. So far, all I have done is gather data. Now I thought it could be interesting to look at this data and see what I've learned.

Before I start I must apologize for any inaccurasies, misspellings and other mistakes, as well as the rambling and incoherent nature of this analysis. I will do my best, but suggestions and constructive critisism is very welcome.

 

Now, onto the numbers!

Sales for the consoles since I started measuring (12 weeks):

Console	Wii	PS3	X360
Total	3,847,616	3,509,679	2,013,190
America	1,711,161	1,227,225	1,137,257
Japan	289,097	555,166	56,284
Total Others	1,847,358	1,727,288	819,649

The weekly average for each console is:
Wii - 320k
PS3 - 290k
Xbox - 170k

These numbers gives us the base against which we can judge the adjustments.

 

Taking the sum of the absolute values of each weeks adjustment gives us the following table:

ABSOLUTE ADJUSTMENTS	Total Adjustments	Average weekly Adjustments
Wii
Total	415 090	34 591
America	35 000	2 917
Japan	6 607	551
Total Others	394 291	32 858
PS3
Total	201 532	16 794
America	71 000	5 917
Japan	83 140	6 928
Total Others	49 836	4 153
X360
Total	213 374	17 781
America	0	0
Japan	2 971	248
Total Others	212 547	17 712

 

To get the cumulative total adjustment for the period you simply add all adjustments together:

CUMULATIVE ADJUSTMENTS
Wii	Total	Weekly Average
Total	246 320	20 527
America	35 000	2 917
Japan	3 793	316
Total Others	207 527	17 294
		0
PS3		0
Total	122 198	10 183
America	71 000	5 917
Japan	17 766	1 481
Total Others	33 432	2 786
		0
X360		0
Total	-181 426	-15 119
America	0	0
Japan	875	73
Total Others	-182 301	-15 192

 

All numbers combined gives us the full picture:

	Average weekly VGChartz current numbers	Average Absolute Adjustments	% Difference	Average Cumulative Adjustments	% Difference
Wii
Total	320 635	34 591	10,79%	20 527	6,40%
America	142 597	2 917	2,05%	2 917	2,05%
Japan	24 091	551	2,29%	316	1,31%
Total Others	153 947	32 858	21,34%	17 294	11,23%
PS3
Total	292 473	16 794	5,74%	10 183	3,48%
America	102 269	5 917	5,79%	5 917	5,79%
Japan	46 264	6 928	14,98%	1 481	3,20%
Total Others	143 941	4 153	2,89%	2 786	1,94%
X360
Total	167 766	17 781	10,60%	-15 119	-9,01%
America	94 771	0	0,00%	0	0,00%
Japan	4 690	248	5,28%	73	1,55%
Total Others	68 304	17 712	25,93%	-15 192	-22,24%

 

So what does this awful table really mean? For the sake of clarity I'll take an example. The Wii has sold an average of 320k per week. During these 12 weeks it been adjusted both up and down by an average of 33k every week, or 21.34% (Absolute adjustment). The result of these adjustments is, in the end, that it lies on average 17k above the originally reported numbers every week, or 11.23% (Cumulative adjustment).

The result I've come up with is this - The originally reported VGChartz hardware numbers are accurate to within +-10%. During these 12 weeks adjustments have indicated that the Xbox is being overtracked by 9%, with all of those numbers coming from others. The PS3 is being undertracked by 3.5%, and the Wii is being undertracked by 11% in others, resulting in a 6.4% overall undertracking.

When looking at the weekly averages for the last 12 weeks and how they have changed, the picture changes quite a bit:

	Average weekly VGChartz numbers	Original weekly VGChartz numbers	% Difference
Wii
Total	320 635	310 864	3,05%
America	142 597	139 680	2,05%
Japan	24 091	23 732	1,49%
Total Others	153 947	147 452	4,22%
PS3
Total	292 473	269 654	7,80%
America	102 269	96 352	5,79%
Japan	46 264	44 798	3,17%
Total Others	143 941	128 504	10,72%
X360
Total	167 766	169 112	-0,80%
America	94 771	94 771	0,00%
Japan	4 690	4 671	0,41%
Total Others	68 304	69 670	-2,00%

The PS3 still has been undertracked significantly in other for this period, but overall percentages is quite predictably down quite a bit compared to the total.

That is, I think, not bad at all. VGChartz is pretty accurate with its weekly figures.

 

What I'm doing here is based on a simple assumption: VGChartz will adjust numbers so that they are accurate. The number of sold consoles in a year is reported, and VGChartz will adjust numbers to reflect those reports. Thus, VGChartz numbers for last year should always be more or less accurate. My goal is therefore to keep track of all adjustments made to previous figures, so that after a while looking at those adjustments can show how accurate the original VGChartz figures were.

Naturally, there are a couple of problems with this analysis. The first problem is that this is all internal VGChartz numbers. I have not gathered data from any other source, so if VGChartz is dead wrong with anything and refuses to adjust it, this analysis cannot catch that. The second problem is that my data is from a relatively short period of time (12 weeks). My numbers will get more accurate over time, but for now the adjustments made every week covers weeks that I do not have data for, which means that the above analysis serves more as a general guideline than anything else. The last problem is that I do not track adjustments made on weekly numbers, but on total cumulative numbers. That means I easily miss subtleties.

I'll try to do another set of numbercrunching when we get to 26 weeks (half a year), but I'm not sure whether or not I'll still be keeping track of this by then. For now, you'll just have to take my figures with a grain of salt.

TL;DR - Pretty accurate (I think)!