In my mind there is no need to try and prove anything to NeoGaf. If it was a matter of questioning how legit the numbers are, maybe I could see it. But it's not about legit numbers, it's a personal and business thing. NeoGaf just won't accept VGChartz and ioi until VGChartz's worldwide acceptance is just to large to resist. Who knows, even then they might be too bitter to accept the facts.

I will give NeoGaf some credit, as the site does have it's moments. Jeff Bell anyone?

http://www.joystiq.com/2007/07/18/jeff-bell-asks-message-board-user-and-your-contribution-to-soc/

Whoa whoa, everyone calm down here.

Apu, thanks for coming over for a bit. Let it be known that I am not one of these "analysts," that I'm just a forum dweller turned mod, and that I am quite familiar with GAF. I am aware of how and why this site is so hated. Please, let me explain why I don't agree, and hopefully we can come to some middle ground here.

When people question the credibility of this site, they always do precisely what you are doing now: pointing to specific instance where the site's data was dramatically off (for a specific game or system in a given month, for example) and needed to be corrected. Please note, as Erik Aston has already pointed out, that we know they're off -- Ioi doesn't scoot this information under the rug. Please check The Source's monthly comparisons to the NPD, where he gives in depth analysis of where VGChartz was off, if it was in error, and why.

So in other words, people like you often come in here and say "look, this figure is wrong!" And our response is "we know." The site averages, on a monthly basis, about 14-15 percent margins of error for all systems, although as the sample time increases, that margin of error decreases significantly, to the point where LTD totals are likely within 2-3 percent of the real totals. Apparently, you view this discrepancy as unacceptable, a sham, or so forth; to us, it's a completely acceptable margin of error and we go in knowing that the monthly sales data is going to be dramatically off on occassion.

Here's another example to highlight the point -- the NPD has a margin of error of about 1-2 percent. Theoretically, someone could insist that this isn't good enough, and that real scientific data would require a margin no greater than .001 percent per system. Any data with a larger confidence interval than that is useless to that person. To you, that person's threshold for valauble data may seem too strict, and I would agree. That's exactly what's happening here, just further down the line. We know there are flaws. We know it's not perfect, and that there are occassionally dramatic errors due to sampling size. We accept it, and recognize that there is still a great deal of value in the data provided. 

 

Bodhesatva said: Whoa whoa, everyone calm down here. Apu, thanks for coming over for a bit. Let it be known that I am not one of these "analysts," that I'm just a forum dweller turned mod, and that I am quite familiar with GAF. I am aware of how and why this site is so hated. Please, let me explain why I don't agree, and hopefully we can come to some middle ground here. When people question the credibility of this site, they always do precisely what you are doing now: pointing to specific instance where the site's data was dramatically off (for a specific game or system in a given month, for example) and needed to be corrected. Please note, as Erik Aston has already pointed out, that we know they're off -- Ioi doesn't scoot this information under the rug. Please check The Source's monthly comparisons to the NPD, where he gives in depth analysis of where VGChartz was off, if it was in error, and why. So in other words, people like you often come in here and say "look this figure is wrong!" And our response is: "we know, the site isn't perfect." The site averages, on a monthly basis, about 14-15 percent margins of error for all systems, and as the sample time increases, that margin of error decreases significantly, to the point where LTD totals are likely within 2-3 percent of the real totals (and shrinking, as this generation goes on). Apparently, you view this discrepancy as unacceptable, a sham, or so forth; to us, it's a completely acceptable margin of error and we go in knowing that the monthly sales data is going to be off on occassion. Here's another example to highlight the point -- the NPD has a margin of error of about 1-2 percent. Theoretically, someone could insist that this isn't good enough, and that real scientific data would require a margin no greater than .001 percent per system. To you, that person's threshold for valauble data may seem to strict, and I would agree. And that's exactly what's happening here, just further down the line.

(emphasis added by me)

Unfortunately, Bodhesatva, unless I'm being completely blind (which is not impossible), that month's NPD/VGC news article did not so much as mention the huge PS2 discrepancy, let alone discuss it or any of the others.  All we got was "As stated in the NPD Preview, Vgchartz Americas data (covering more than just the USA) had the following totals most of which are similar to NPD USA data:" 

Obviously "most" being "similar" allows for one to be drastically dissimilar, but that's hardly the same as "in-depth analysis". 

He could've not put in the table at all?

PS2 was very different but it's really the NPD numbers that are odd in that case. They claim that PS2 sales dropped by half weekly from Feb to March for no reason.

Bodhesatva said: Whoa whoa, everyone calm down here. Apu, thanks for coming over for a bit. Let it be known that I am not one of these "analysts," that I'm just a forum dweller turned mod, and that I am quite familiar with GAF. I am aware of how and why this site is so hated. Please, let me explain why I don't agree, and hopefully we can come to some middle ground here. When people question the credibility of this site, they always do precisely what you are doing now: pointing to specific instance where the site's data was dramatically off (for a specific game or system in a given month, for example) and needed to be corrected. Please note, as Erik Aston has already pointed out, that we know they're off -- Ioi doesn't scoot this information under the rug. Please check The Source's monthly comparisons to the NPD, where he gives in depth analysis of where VGChartz was off, if it was in error, and why. So in other words, people like you often come in here and say "look, this figure is wrong!" And our response is "we know." The site averages, on a monthly basis, about 14-15 percent margins of error for all systems, although as the sample time increases, that margin of error decreases significantly, to the point where LTD totals are likely within 2-3 percent of the real totals. Apparently, you view this discrepancy as unacceptable, a sham, or so forth; to us, it's a completely acceptable margin of error and we go in knowing that the monthly sales data is going to be dramatically off on occassion. Here's another example to highlight the point -- the NPD has a margin of error of about 1-2 percent. Theoretically, someone could insist that this isn't good enough, and that real scientific data would require a margin no greater than .001 percent per system. Any data with a larger confidence interval than that is useless to that person. To you, that person's threshold for valauble data may seem too strict, and I would agree. That's exactly what's happening here, just further down the line. We know there are flaws. We know it's not perfect, and that there are occassionally dramatic errors due to sampling size. We accept it, and recognize that there is still a great deal of value in the data provided.

Actually no. NPD uses 95 percent confidence interval (as stated by NPD, and which is the most commonly used), which gives 5% margin of error. So, even wanting the numbers to be in 1-2% error margin would be calling foul on NPD numbers.

1% - 10% margin error is usually acceptable in industry practice. When a company state that a transistor has a resistance of 10K ohm, as long as the actual value fall in 9 - 11 K ohm range, it is deemed acceptable. Other industry might not accept 10% error, and can only accept 1% error. Yet other industry, such like Integrated Circuit, might want to achieve Six Sigma, 1 part per million error.

The problem is this : Without NPD data as guidance, your weekly data will be way off. Of course, your LTD data could have an insignificant error, since you have console manufacturer shipment data as benchmark.

The major value of your site (IMO) is the weekly data. If the weekly data is wrong, then the LTD will also be wrong (unless you have external benchmark, be it NPD, Famitsu / M-create, ELSPA, company production shipment).

For S/W LTD, that is where the most discrepancy came from. Unless you assume that shipment = sold (which I don't agree with), and you have data of S/W shipment, your S/W LTD are most probably very inaccurate.

Is there a clear indication in the database of which S/W have high confidence (because there is shipment data available), and which one have low confidence (no shipment data) ? If there is no indication at all, how can we be confident that we are talking about an amount that is roughly correct (+- 10% deviation) ?

You should ask yourself :

• Jan 08, PS2 : 4 weeks : 264K
• Feb 08, PS2 : 4 weeks : 351K
• Mar 08, PS2 : 5 weeks : 216K

Normally, old console (last gen console) keep on trending lower each and every month (other than holiday month, like Nov and Dec).

The real question is : What happen on Feb to cause such a sharp increase in PS2 sales ? March is only a return to normalcy for PS2 (lower weekly # compared to Jan & Feb).

apujanata said: So in other words, people like you often come in here and say "look, this figure is wrong!" And our response is "we know." The site averages, on a monthly basis, about 14-15 percent margins of error for all systems, although as the sample time increases, that margin of error decreases significantly, to the point where LTD totals are likely within 2-3 percent of the real totals. Apparently, you view this discrepancy as unacceptable, a sham, or so forth; to us, it's a completely acceptable margin of error and we go in knowing that the monthly sales data is going to be dramatically off on occassion. Here's another example to highlight the point -- the NPD has a margin of error of about 1-2 percent. Theoretically, someone could insist that this isn't good enough, and that real scientific data would require a margin no greater than .001 percent per system. Any data with a larger confidence interval than that is useless to that person. To you, that person's threshold for valauble data may seem too strict, and I would agree. That's exactly what's happening here, just further down the line. We know there are flaws. We know it's not perfect, and that there are occassionally dramatic errors due to sampling size. We accept it, and recognize that there is still a great deal of value in the data provided. 1% - 10% margin error is usually acceptable in industry practice. When a company state that a transistor has a resistance of 10K ohm, as long as the actual value fall in 9 - 11 K ohm range, it is deemed acceptable. Other industry might not accept 10% error, and can only accept 1% error. Yet other industry, such like Integrated Circuit, might want to achieve Six Sigma, 1 part per million error. The problem is this : Without NPD data as guidance, your weekly data will be way off. Of course, your LTD data could have an insignificant error, since you have console manufacturer shipment data as benchmark. The major value of your site (IMO) is the weekly data. If the weekly data is wrong, then the LTD will also be wrong (unless you have external benchmark, be it NPD, Famitsu / M-create, ELSPA, company production shipment). For S/W LTD, that is where the most discrepancy came from. Unless you assume that shipment = sold (which I don't agree with), and you have data of S/W shipment, your S/W LTD are most probably very inaccurate. Is there a clear indication in the database of which S/W have high confidence (because there is shipment data available), and which one have low confidence (no shipment data) ? If there is no indication at all, how can we be confident that we are talking about an amount that is roughly correct (+- 10% deviation) ?

Yes and no. There's only a number of units the channel can take. Of course, there are some exceptions in channel stuffing, but this will be indicated by next quarters low shipments. And in the other hand, a normal situation without supply constrains, the percentage of change to the previous quarter, doesn't vary much from the competition, so out of the 3, the one that's stuffing, should be easilly identified. And in the other hand, changes in comparision to historical data should also help to identify this.

Now, the SW is the hardest part, but in the other hand, having a "close enough" estimation is a lot better than having no data at all. And again, you can get pretty close by knowing the number of shipments.

DKII said: He could've not put in the table at all? PS2 was very different but it's really the NPD numbers that are odd in that case. They claim that PS2 sales dropped by half weekly from Feb to March for no reason.

Rebuttal A: I do appreciate the fact that we have the numbers in the first place, but the fact remains that there was no analysis of the NPD-VGC differences. None. At. All. So saying a number-dump = "in-depth analysis" is incorrect.

Rebuttal B: NPD has March lower than Feb for no reason? Surely VGC would never do that.

(The "month" in question, of course, starts the week after the spike.)

1% - 10% margin error is usually acceptable in industry practice. When a company state that a transistor has a resistance of 10K ohm, as long as the actual value fall in 9 - 11 K ohm range, it is deemed acceptable. Other industry might not accept 10% error, and can only accept 1% error. Yet other industry, such like Integrated Circuit, might want to achieve Six Sigma, 1 part per million error.

 

For what it's worth, resistors have 20% tolerances too, it's not that uncommon:

http://en.wikipedia.org/wiki/Electronic_color_code

Different applications have difference tolerance requirements.  For a hobby site that counts weekly sold consoles and gets adjusted based on shipment data, I don't see a  real problem with 20% tolerance.  It's certainly not such a huge issue that people have to get as worked up about it as NeoGAF has, especially if the error averages out over time.