Are you using a new model of the Vita? I only have the original OLED one and I can't imagine how you could accidentally move the memory card. In mine its inside the Vita protected by a little door (and then I also have a L2/R2 grip case thing over the top) I've removed the card before too and all it did was pull everything out of my folders.
It does suck having to download everything though, I'm currently downloading and backing everything up on my computer which is taking forever but it should save time since I constantly have to delete games to make room for new ones and then redownloading can take much longer than simply copying from my HDD.







