By using this site, you agree to our Privacy Policy and our Terms of Use. Close

Forums - Website Topics - Ruh-roh: Simon Carless of Gamasutra/Game Set Watch analyzes VGChartz...

Just_Ben on 24 June 2008
ioi said:

@Just_Ben, who's being childish now?

I actually like the idea of putting together a margin of error statement for the site. My issue is that the only way to do it is experimentally using some of the data we post and basically compiling and comparing the accuracies. The second problem is that the only data we have to compare to is NPD, which I feel has reasonable errors in its own data. So unless we can somehow setup a random sample of data and find corresponding "real" data to do a study against (a third party can do this if they wish) then I struggle to see how we can do it. And it's certainly not something that can be demonstrated analytically. If a publisher wanted to partake in a study to compare their shipment and retailer feedback data to ours so that we can say "90% of the time we are within 10% of the real figures" or whatever I'd be happy!!

Me? Yes that was childish, but there is a difference between you and me. And as soon you would set me in an official position for your site, I would stop doing that. But since you and me agree that I don't fit on any off the positions you could offer, I have the freedem to be so ;)

 



Join up to the VGC Mario Kart Wii GP European League!

Or the VGC Mario Kart Wii GP Australian League!

Or the VGC Mario Kart Wii GP American League!

Or the VGC Halo 3 American League!

Or the  VGC Call Of Duty PS3 League!

Around the Network
alpha_dk on 24 June 2008
ioi said:

@Just_Ben, who's being childish now?

I actually like the idea of putting together a margin of error statement for the site. My issue is that the only way to do it is experimentally using some of the data we post and basically compiling and comparing the accuracies. The second problem is that the only data we have to compare to is NPD, which I feel has reasonable errors in its own data. So unless we can somehow setup a random sample of data and find corresponding "real" data to do a study against (a third party can do this if they wish) then I struggle to see how we can do it. And it's certainly not something that can be demonstrated analytically. If a publisher wanted to partake in a study to compare their shipment and retailer feedback data to ours so that we can say "90% of the time we are within 10% of the real figures" or whatever I'd be happy!!

I can't really think of any good ways to do it either; if we only use titles that we have numbers for in press releases, end of year statements, etc, then we are just going to be matching data that we already match

However, if we just take a random sample of games on this site, we may easily get to the point where 50% of the games we're trying to match up to, there is no public data about.  It's a conundrum to say the least; I've been trying to think of an analytical, substantative method to estimate this since this thread started, and I can't think of anything that will both be possible and unbiased, short of taking a random sample of our data and asking the publishers their shipment numbers for those titles if they haven't released them yet.  Unfortunately, this would require contacts in the industry I don't have, and even with those they're likely to say 'no.'

All told, I am not sure that I can think of a way to estimate it in such a way that it will likely happen, short of broad generalizations like I posted above; 'unconfirmed by reports' 'confirmed by reports', etc.

That would at least give a very broad level of accuracy to the data; I would prefer something a bit more precise, but that may not be possible without additional help from publishers/distributors/whomever.



Please, PLEASE do NOT feed the trolls.
fksumot tag: "Sheik had to become a man to be useful. Or less useful. Might depend if you're bi."

--Predictions--
1) WiiFit will outsell the pokemans.
  Current Status: 2009.01.10 70k till PKMN Yellow (Passed: Emerald, Crystal, FR/LG)

JiaJia on 24 June 2008

I agree with the article but then again what other choice do we have for sales figures updated weekly? ... Inclusing Europe as well ... =P



shio on 24 June 2008
steven787 said:
theRepublic said:
shio said:
steven787 said:
Accuracy of 10 or even 20 percent error isn't a big deal when you are discussing Qualitative properties of sales. Of course publishers and retailers want more accurate data, but as far as using sales to predict future trends in gaming (what type of games sell, what console is "winning") broader estimate work just fine. I don't see what the problem is or why the established game websites are so against the idea of VGChartz.

a 10% error, let alone 20%,  is a mortal flaw in methodology. Any professional outlet that often possesses that terrible margin of error wouldn't last long in the business world.

The article didn't say much I didn't already know, and I agree completely with it.

 

To Brett Walton: Please give credit to your sources, that's the way it should be.

 

Read this post from naznatips:

"That said, all data trackers provide nothing but estimates. ALL. Famitsu and Media Create differ as much as 40% on a given week, and each one tracks more of the Japanese market than NPD tracks of the NA market. Pretending any sales data gives anything more than a basic idea of sales trends is laughable."

Famitsu and Media Create are profesional tracking services.  I guess they and the NPD are all mortally flawed by your standards.

This is not science, and it is not a labratory.  There are different standards in the real world.

 

Shio, your missing the point.  If you want more accurate data, you are (1) not going to get it and (2) setting your standards unecesarily high.  We are just here having fun, talking about general trends, etc.

Common anti-social thinking.  People are having fun, oh my god, I need to stop it.

 

VGChartz is free.

VGChartz is open about its errors and goes out of its way to compare their results to non-free trackers.

VGChartz is fun.

Next the false logic in this thread is really nagging me:

Non-free trackers aren't necesarily better.  One of the first things you learn in (a real) statistics class is that a larger sample can mess up extrapolation.

I didn't miss the point. As I said before, I know what VGChartz is all about.

The problem is that I woudn't like if VGChartz takes data from Media Create, NPD, etc... and not give credit to them. It is simply unethical.

 



http://i48.tinypic.com/120qaon.jpg
darthdevidem01 on 24 June 2008

Not a bad article....it defineely does CherryPick.,..but the word that VG Chartz exists is getting round n thats important



All hail the KING, Andrespetmonkey

Around the Network
Retrasado on 25 June 2008

@ioi
why don't you post the full text of your email to that guy here?

btw, I love this site....



Not trying to be a fanboy. Of course, it's hard when you own the best console eve... dang it

Renar on 25 June 2008

ioi,

It's a bit sad of a commentary on journalism, but it does seem that, unless you know and trust the journalist or the place that they work for, you need to assume that they are equivalent to National Enquirer. And that any restrictions that you would put on the publishing of your comments, you would want the wording to stand up in a court of law as basically as 'contract language'. "Appreciate", IMO, is too 'nice' of a word. "I require that you", sadly, would have been better.


To others on another topic brought up here:
And as far as the stock market is concerned, high priced analysts from very respectable firms get earnings estimates wrong all the time. I could point out a number of quarterly earnings that are off by large percentages, especially if the earnings amounts per share are small to begin with. So, it shouldn't be surprising that lower volume games are off more %-wise as well.



Torturing the numbers.  Hear them scream.