Singularity
Timeshift
Henry Hatsworth in the Puzzling Adventure
Dead Space Extraction
Deadly Creatures

Pandora's Tower, OpRainfall or otherwise, this was an under appreciated gem.

the game that came to my mind the moment i read this threads title was Vanquish, so im gald to see it was the first game to be mentioned in the whole thread.

aside from that Valkyria Chronicles is debatable, since it used to be a overlooked gem, but with time it has become pretty popular among "hardcore" gamers, to the point that you see it mentioned in a LOT of forums and gaming websites, so it must have mainteined some good sales through the years to the point that i dont think you can say its an overlooked gem anymore.

EDIT: just searched its sales and Valkyria Chronicles sold 1.19 million copies... so i truly dont know if those are good numbers or not, for this type of game.

The Rune Factory games..... those are real good.
Red Steel 2 was a great game that should be played by all.
Ninja Gaiden Dragon Sword is criminally over-looked and uses the DS in book style.
Picross 3D is my favorite puzzle game of last gen (needs a 3DS sequel.)
Excite bots is the best arcade racer of last gen (hands down.)

Those are just of the top of my head but yeah all of those are amazing gems.

It is sales wise even though it might be undertracked a bit, but even if numbers where 100% accurate it probably still would be well under 1 million copies sold.  Also its been out of print for a while now since new unopened copies are now getting harder to find. 

Rapala Pro Bass Fishing 2010
Hellboy: The Science of Evil
Dark Void
Akai Katana
Otomedius Excellent
Deathsmiles
Corpse Party
Spider-Man 3
Turning Point: Fall of Liberty
Aquapazza: Aquaplus Dream Match
Juiced: Eliminator
Full Auto 2: Battlelines
Anarchy Reigns
Blazblue (all 3)
Condemned 2: Bloodshot
FaceBreaker
Fantastic Four: Rise of the Silver Surfer
Genji: Days of the  Blade
Kung Fu Rider
Shaun White Skateboarding

There was a thread not too long ago about games that surprised you with awesomeness.  With a few tweaks, my list for that is pretty much the same as overlooked gems.

Art Style: ORBIENT - This is such a simple game, but it was completely addicting.  All you do is play around with gravity.  You play as a star, and you only have two control options, 'A' for attract and 'B' for repel.  You absorb smaller stars to grow in size or catch them in your orbit to score points, all while avoiding crashing into larger objects or getting sucked in by black holes.

de Blob - One of the best 3D platformers I have played not named Mario.  Who knew painting could be so fun?  Great use of visuals and music.  It is very rewarding to turn the completely white environments into a rainbow of colors.

Fluidity - Fun little puzzle platformer.  You play as a puddle of water.  Yes, a puddle of water.  You use tilt controls to move the water around.  Eventually you get new powers, like turning into a block of ice or becoming a cloud of steam.

LostWinds: Winter of the Melodias - This one is an exploration based platformer.  It makes great use of the Wii remote IR pointer to control a wind spirit that helps the little boy you control with the analog stick.  The first was good, but the sequel felt like a big improvement.

Muramasa: The Demon Blade - I wasn't too sure what to expect of this one.  I heard it was good, so I gave it a try.  It turns out it is just a fantastically well made, high-quality game.  Beautiful visuals.  Amazing boss battles.  Tons of fun.

If it wasn't already mentioned... Madworld for Wii!

Valkyria Chronicles was the MAIN reason I bought a PS3. I actually bought the game more than 6 months before the console as it was hard to find.

In Japan it debuted @ #3-78,000, to #10-15-16- to #29 with a little more than 120.000, in 6 years it has doubled that amount which is rare for a non Nintendo game.

In Europe, it never did make the chart.

In NA, it had one week on the chart @ #35, then dropped off the chart, yet still managed to sell 670,000 lifetime. It's legs were due to word of mouth, and Sega constantly releasing more copies as more and more gamers turned others on to this fine game. Most of us were amazed it even hit a million, not that it didn't deserve it, just amazed in how long it took to get there.

A game not mentioned yet is Opoona, a unique  labour of love RPG, that must of turned off many gamers, with it's confusing as all get out areas.

Condemned criminal orgins was fantastic as well as Blue Dragon and Lost Odyssey