3DS by far. Best platform for JRPGs at the moment

If you're a huge RPG fan, you're going to have to get both eventually. I'd say 3DS has the edge though. 3DS + DS backwards compatibility + Virtual Console = win!

As was already stated, the Vita can play PSP and PSX games so long as they've got digital versions and the 3DS can play all DS games.

When it comes to exclusives, the 3DS has the Vita beat in quality. SMT4, Devil Summoner Soul Hackers, Bravely Default, Fire Emblem, Etrian Odyssey 4, Etrian Odyssey remake, Heroes of Ruin, and Bravely Default are good. Soul Hackers isn't exactly exclusive, but it is in the US. For the Vita when it comes to exclusive RPGs, there's Ys Memories of Celceta which is a good action RPG and Curse of the Curry God which is a rogue like RPG game.  I haven't played the rogue like, but very rarely do rogue like games suck.

When it comes to quality whether it's ports and backwards compatibility, then the Vita takes it.

The 3DS can play all DS games, so you've got Contact, Chrono Trigger, Dragon Ball Z Attack of the Saiyans, Dragon Quest series, Final Fantasy series, Fire Emblem, Front Mission, Golden Sun, SMT Strange Journey, Suikoden Tierkreis, Retro Game Challenge(a good old school RPG and good non RPG games,) and others.

The Vita has Vita games like Persona 4 Dragon's Crown,and the Atelier series, then it's got the PSP and PSX library. These are PSP games I'd recommend that can be downloaded directly from the PSN store. Crimson Gem Saga, Final Fantasy 4 Complete Collection, Growlanser Wayfarer of Time, Half Minute Hero, Persona series(under SMT,) Sega Genesis Collection(Phantasy Star 2-4, Sword of Vermillion, and a lot of good non RPGs,) Untold Legends series, Wild ARMs XF, Yggdra Union, and Ys series.

Here are the PSX games I'd recommend that can be downloaded. Persona 2 EP(sequel of the PSP game,) Square Enix games, Vanguard Bandits, Breath of Fire 4, Grandia, Arc the Lad 1, 2, 3, Arc Arena, Wild ARMs 1, and Wild ARMs 2.

I do not have a Vita so I have no opinion there. The 3DS however has some truly amazing RPGs. EO3, Shin megami tensei IV, mario & luigi RPG and the whole DS list of stunning Rpgs. DQ9, TWEWY, GOlden Sun, Valkyrie profile and many more.

I might be a bit bias, but I would go with the Vita. Persona 4, Ys Celceta, Final Fantasy X/X-2 Remake, Soul Sacrifice, and tons more, are all great on the Vita. A lot of games are also finally coming over from Japan that you may like, and you also have that huge backlog of the great PSP and PS1 jrpgs.

The PSP blows away both of them, amazing titles!

Jeanne Dark, Valkyria Chronicles 2, Persona 3, Star Ocean First Departure, One minute hero, Class of Heroes, Final Fantasy 1 and 2, and hundreds of others

Thanks for the info, all. I think I'll end up going with the Vita, but only because of backwards compatibility with PSP/PSX games as I have played exactly zero of those games. I was an N64 guy so I never played any PS exclusives back then, and I also never owned a psp.

It seems like the Vita will give me a ton of choices for games I should have played, but never ended up getting a chance to.

You'll love it, the Vita's a great console. Its library of RPGs is growing every week, and I have no doubt that it will one day come close or even match the PSP in terms of quality RPGs. 

I don't think that works on Vita o.o (and it was pretty rubbish compared to the first)

Love the games, mostly because the builds are so broken and OP