Success is when a game meets goals, and it can be represented in multiple ways. With some games, there is no clear cut solution.
Fire Emblem Shadow Dragon is a success because even though it only sold 500k, it outsold every other game in the franchise, and got great reviews.
Enslaved is not a success because Ninja Theory have been struggling to meet their financial goals with their games, and Enslaved did not meet their financial goals. Although ratings are good, it's clear Ninja Theory cared about sales more (at least to me, hence success is independent on perspective).
Castlevania and Bayonetta sold a modest amount of units, but due to their high scores, and well received reviews, they are still very successful games.
Lost Odyssey is a success for gamers (it's a great RPG), but not a success for Mistwalker as the game only sold 600k. I'm pretty sure they intended to make that franchise a Final Fantasy compeditor (similar name, FF was PS3 exclusive at the time, creator of FF made Mistwalker).
Then games like CoD are success's based on how much people play online, number of unique users, and number of copies sold.
So it really depends on the game, and for most games it is a mixture of both. Developers have a target they want to meet (say 500k units per console for Vanquish, or 12 million for CoD), and if it meets that financial target it's a success. Developers also want to meet a critic goal (say a certain rating, or level of quality). Generally the more shocking of the two is the one which determines if a game is a success or not, but usually the most shocking games are success's/failures as a mix of both.
E.G. Vanquish is selling better then Sega expected, and received higher ratings then most people expected, and is just a darn fun game to play. Hence it's a critical and financial success.
GT5 is selling well financially, but in terms of game quality, it's below what many expected. So it's a mixed bag.
FFXIII is also a mixed bag. Sales are pretty strong, combined sales are at the level FFXII had. However western reviews gave the game on average an 8.5/10 due to linearity, which is very poor for a game this established.
However in the end, I only car about quality, how fun a game is, and replay value. I'm really enjoying Vanquish, Dead Space 2, Forza 3, GT5, GTA IV, and Assassin's Creed: Brotherhood right now, and they all come from a different target groups (GTA is mega high, Dead Space 2 is average, and Vanquish is niche).