Well, Pezus already posted the complete list, so I will just add some advices. Get a big memory card, at least 32 GB. My Vita came with a 4 GB one and I bought a 8 GB card at the same time to replace it. I got PS All Stars bundled (digital) so when I got plus the card wouldn't store more than one game (I had a PS1 title too). Of course, you can back up games on a PS3 or PC and only keep on the card the ones you're playing, with saves stored on the PS Plus cloud.
My new 32 GB card made everything easier. I have all games of Pezus' list installed, except Gravity Rush and Uncharted because I got them at retail with my Vita. All of them are pretty good, I'm having a good time with Soul Sacrifice (nice online coop mode) and Sonic All Stars. Oddworld surprised me too and is pretty good. Do you have a PS3 too? Plus is killing on it too.