By using this site, you agree to our Privacy Policy and our Terms of Use. Close

Forums - Gaming - Why is the Review System Broken?

One broken element about it is the way it is scored by most sites: where a game is broken down into component parts each part getting a score then those scores being averaged. It is an attempt at objectivity when reviews are inherently subjective.

No other entertinment reviewing system employs reductionist scoring. Some review sites are eliminting the reductionist method and either giving one score for the review, or doing reductionist scoring but giving an overall score which is not simply an average of the component scores.

I think VGC should get rid of reductionist scoring, or at least not base the overall scoree of a game in the average of the 3 scoring elements. To prevent people from complaining that the overall score doesn't match the component score averages I think the reductionist scores should be on the ABC scale (with no such thing as greater than A+), and the final score be /10 or /100.

The other problem with reductionists scoring is that it tends to overemphasise negligible differences between games. Is there really a difference between a 9.5 and 9.6 game? No there isn't, but the people who prefer the game that got the 9.6 and don't like the game that got the 9.5 will make a big deal over that 0.1 difference.

Games scored on a 1-10 scale should have 0.5 point graduations and nothing more fine than that. If a reviewer with their own assessment system arrives at a X.2 or X.7 result they should use their own personal opinion about whether to move the score up or down. No such thing as automatically rounding up or down. even a game that rates 8.1 on the reviewers own system could put it up to 8.5 basically because they definitely think it's better than the 8.0 games they've reviewed and it sits better in their personal pool of 8.5 games. Or an 8.9 game definitely doesn't warrant 9.0 in the reviewers OPINION so it gets knocked back to 8.5.

I think one of the issues with game review scoring is that I am happy to pay $15 (NZD) to see a 3-star movie (6/10) at the cinema if it's a type of movie I like, and I'll watch crappy 1-star movies for free if I've got nothing better to do. But if I have to spend $50 or worse $100 on a game, or rent it multiple times in order to finish it then it's really only the 8+ and in many cases the 9+ games that are going to justify taking a risk with that sort of money.

PSN/XBL/WiiWare games that are only $10-$15 are worth getting even if only scoring 6-7/10 if they are in the right genre. Because the amount paid is comparable to the level of enjoyment.

Another comparison is: Movies that review badly can be box office smashes. Like Transformers 2. It got panned by reviewers yet it is one of the biggest grossing movies of all time in the USA. How many games that received low reviews have really high sales? None I'd wager, at least not within the last 10 years. The fact is much much moreso than any other forms of mass market entertainment, review scores = sales potential for games. About the only other entertaiment where reviews = life or death for the production is in live theatre.

With that kind of pressure it is only natural that the gaming industry will be gunning for ever higher review scores. Same as live theatre where producers are absolutely desperate for glowing reviews from the most respected reviewers.



“The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt.” - Bertrand Russell

"When the power of love overcomes the love of power, the world will know peace."

Jimi Hendrix

 

Around the Network

I don't really see the reviews as better or worse now than before, I've always subscribed to gaming magazines (when such things were hot) and paid a lot of attention online.
The main difference now is that the audience (us) whine a whole lot more about the scores.
Don't take this the wrong way, I whine as well but mostly about games that get undeserved (imo) high scores.
I don't really think the gaming press as a whole has a bias against any one console but they do seem to treat certain developers more kindly.



btw,


PS2 = 63 GAMES WITH 90 SCORES

XBOX= 31

GAMECUBE= 26


VS


PS3 = 16

360= 24

WII= 9


Now, how is it any different than last Gen? except the fact reviewers are not given Wii games high scores.



 Next Gen 

11/20/09 04:25 makingmusic476 Warning Other (Your avatar is borderline NSFW. Please keep it for as long as possible.)
binary solo said:
One broken element about it is the way it is scored by most sites: where a game is broken down into component parts each part getting a score then those scores being averaged. It is an attempt at objectivity when reviews are inherently subjective.

No other entertinment reviewing system employs reductionist scoring. Some review sites are eliminting the reductionist method and either giving one score for the review, or doing reductionist scoring but giving an overall score which is not simply an average of the component scores.

I think VGC should get rid of reductionist scoring, or at least not base the overall scoree of a game in the average of the 3 scoring elements. To prevent people from complaining that the overall score doesn't match the component score averages I think the reductionist scores should be on the ABC scale (with no such thing as greater than A+), and the final score be /10 or /100.

The other problem with reductionists scoring is that it tends to overemphasise negligible differences between games. Is there really a difference between a 9.5 and 9.6 game? No there isn't, but the people who prefer the game that got the 9.6 and don't like the game that got the 9.5 will make a big deal over that 0.1 difference.

Games scored on a 1-10 scale should have 0.5 point graduations and nothing more fine than that. If a reviewer with their own assessment system arrives at a X.2 or X.7 result they should use their own personal opinion about whether to move the score up or down. No such thing as automatically rounding up or down. even a game that rates 8.1 on the reviewers own system could put it up to 8.5 basically because they definitely think it's better than the 8.0 games they've reviewed and it sits better in their personal pool of 8.5 games. Or an 8.9 game definitely doesn't warrant 9.0 in the reviewers OPINION so it gets knocked back to 8.5.

I think one of the issues with game review scoring is that I am happy to pay $15 (NZD) to see a 3-star movie (6/10) at the cinema if it's a type of movie I like, and I'll watch crappy 1-star movies for free if I've got nothing better to do. But if I have to spend $50 or worse $100 on a game, or rent it multiple times in order to finish it then it's really only the 8+ and in many cases the 9+ games that are going to justify taking a risk with that sort of money.

PSN/XBL/WiiWare games that are only $10-$15 are worth getting even if only scoring 6-7/10 if they are in the right genre. Because the amount paid is comparable to the level of enjoyment.

Another comparison is: Movies that review badly can be box office smashes. Like Transformers 2. It got panned by reviewers yet it is one of the biggest grossing movies of all time in the USA. How many games that received low reviews have really high sales? None I'd wager, at least not within the last 10 years. The fact is much much moreso than any other forms of mass market entertainment, review scores = sales potential for games. About the only other entertaiment where reviews = life or death for the production is in live theatre.


With that kind of pressure it is only natural that the gaming industry will be gunning for ever higher review scores. Same as live theatre where producers are absolutely desperate for glowing reviews from the most respected reviewers.

I was going to say that was the issue.  Transformers 2 was successful at box office because while reviewers panned the movie for poor story, plot holes, and little substance, people weren't expecting any of those things in the movie and were only expecting robot on robot fighting and that's exactly what they got.  In fact, I can name a few movies that got great reviews but flopped at box office.  A good example is The Iron Giant which got near universal praise and it was an excellent movie IMO but flopped at box office though it did well on DVD from my understanding.



The top selling games of this generation hardly got amazing reviews.

So no.

And the best performing movie of 2009 was Harry Potter and the Half Blood Prince, which got great reviews.



 

 

Around the Network

If reviews mean 'is this game fun?' which is what they currently seem to mean and reviews show a strong correlation to sales with 90+ games on metacritic being almost exclusively multimillion sellers for the HD consoles why should we change to a system where the reviewers don't give a crap what the average movie goer wants?

It seems to work pretty well.

Orange box
Street Fighter IV
Batman AA
Forza 3 (will be soon)
Fifa 10 (maybe)

out of 12 90+ full releases only 25% of them will fail to sell 2M+ on the Xbox 360.



Tease.

One problem I have is that it does seem to be terribly hype driven. It seems as though certain games are assumed to be AAA status almost from their inception, and then with the hype it seems to become a self fulfilling prophecy.

I also agree with the point that there is simply too much weight given to the score of a game a very little to the actual review with scores subsequently used for bragging rights.

A score on a hundred point scale (or 10 point scale with 0.1 increments) implies that somehow reviewers are able to objectively quantify the quality of the game with no bias, it's just not possible. It's an opinion formed through playing the game and comparing what it has to offer with other games on the same system, in the same genre etc. I think a 5 star or 10 point scale (with 1 point increments) with a well written review should be more than enough to tell people how likely it is they are going to like a particular game.

Following on from this the idea that there is a certain score that a game "deserves", because it apparently something we can objectively quantify means that anyone who dares to score below what the game 'deserves' risks the fearsome wrath of the fans caught up in the hype.

p.s. I also found it interesting that people think of 70% in school being a bad score. Perhaps we have alot of good students on this site but generally in school in Australia (at least when i was there), 50% is the pass mark for most subjects and there is no way the class average would be anywhere near 70% for any subject.



Game reviews are nothing more than marketing tools to promote popular titles. Very rarelly do the games justify the scores they receive from game reviewers.
More hype, publicity and popularity = higher review scores.
People rarely read the full game review they usually just read the final score.



numonex said:
Game reviews are nothing more than marketing tools to promote popular titles. Very rarelly do the games justify the scores they receive from game reviewers.
More hype, publicity and popularity = higher review scores.

People rarely read the full game review they usually just read the final score.

Explain Assassin's Creed because while some reviews went easy on it, others were quite harsh towards it.



sc94597 said:
My main problem with the review system is the 7-10 scale opposed to the 1-10 scale. I've noticed that every game that is "worth getting" is from 7-10 and everything else goes somewhere under that. Because of this way of going about things, there is only like 3 areas you can go in. 7-7.9 8-8.9 and 9-10. So things start to get clumped together and games I would rate a 9 are in the same category as games I wouldn't and vice-versa. The games aren't actually rated from 1-10.

The "7-10" system isn't a bad thing though because a 7-10 should be a good game while things below it just get progressively worse.

That's the way things should be.

The problem is though, now, if it isn't an 8 it's terrible and if it isn't a 9 it isn't good.  I've actually seen people say they only play 8.5+ games like anything below that is terrible.  That shit is just ridiculous.

I don't remember this being a huge problem last gen, only when Metacritic and Gamerankings became big.  Of course publishers have always done all they can to get the highest scores, but this gen, I don't know, it just seems so much worse.