Whether you dump the Vita depends on whether you want to educate yourself better on what it actually offers. I would ask you first a few questions and will calrify some of the issues you raised:
1. No. The Vita doesn't only just have Killzone Mercenary, Little Big Planet and Tearaway. It also has these:
Uncharted Golden Abyss
Gravity Rush
Virtue's Last Reward
Soul Sacrifice
Muramasa Rebirth
Assassin's Creed Liberation
YS Memories of Celcetta
Persona 4 Golden (not just a port, it has a ton of extra content)
Those are 7 awesome games that are just as good as the ones you mention. On another level there are other titles such:
Mortal Kombat (with a ton of new exclusive content), Resistance Burning Skies, Smart As, Ninja Gaiden Sigma Plus (with new content), Ragnarok Oddyssey, etc...In other words plenty of Vita games.
2. It is up to you if you want to play on the go the same console versions of Mortal Kombat, Rayman Legends, The Walking Dead, Ninja Gaiden Sigma PLus 2, Dead or Alive 5 Plus, as soon Borderlands 2 and Minecraft. If you value playing any of these games on the go it is worth considering to keep.
3. I don't know if you ever played the gems of the PSP and if you are aware that most of them are accessible on the Vita through the PSN store. These include: The two God of Wars, Ratchet and Clank Size Matters, Final Fantasy Dissidia, Parasite Eve 3, Metal Gear Solid Portable Ops, Valkyria Chronicles 2, Grand Theft Auto Liberty and Vice City Stories, Patapon 1-3, Gran Turismo PSP, Little Big Planet PSP, Resistance Retribution, Syphon Filters, etc... I never played many of these and they are great games as I am discovering. There is an entire library.
4. The PS Vita has most of the relevant indie games coming out on it. For whatever that is worth. Minecraft and Fez are coming soon as well. It's cool to play these games remotely on a smaller screen, many feel better with the many options of control that the Vita gives you.
5. Playstation Now. This alone should be a system seller. To be able to play games like God of War Ascension, Beyond Two Souls, and especially the Last of Us couldn't be cooler.
Beyond these 5 reasons, it's up to you. I think it is a killer system with some great exclusives. They are not just 3, add my 8 and it makes at least 11. 11 must play games plus most of the PSP and PS One library on the store. It will still have more exclusives as some are on their way soon like Dangaronpa and Toukiden. I am absolutely thrilled with my Vita. If I combine its exclusives plus its PSP library, I think it is reason enough to own one. Then PS plus and Playstation Now are even further reasons to keep it. The 3DS is a great system but that doesn't mean that the Vita isn't. Sales aren't everything. Wait for E3. There will be more Vita announcements, it is selling a lot better than it used to.
