3DS will win similarly to how DS won against the PSP. The Vita (Sounds like Pita) is a very big portable device, and is not going to fit in peoples pockets like the 3DS can. I think this is a very major reason why the PSP might have sold decently hardwarewise, but failed miserably on the software front - the thing is, Vita is even bigger than PSP. It is easy to carry both a 3DS and an iPhone, but there is no place for a Vita.
I think the Wii U is going to be the big competitor against Vita, and that 3DS will be in a category of its own. The Wii U controller and Vita will both be systems that will be portable and stay at home. The difference is that the Wii U controller is larger, lighter, streams from a far more powerful device; it essentially destroys the Vita in all the categories you are comparing the 3DS and Vita in.
Another factor that people may have forgot is that Nintendo has the option to drop the 3DS price down when they want to, that option is not available to Sony, as that large machine has a expensive manufacturing cost.
The Vita also does not have the games that Nintendo has - it has games that are essentially updates of PSP titles and down-ports of PS3 titles; Nintendo has games like Mario Kart, Animal Crossing, Mario, Pokemon, and many more; all of which are proven to be able to sell 10 million+ on handheld systems. The best title Vita has is Monster Hunter, and that probably has potential for up to 5 million.
I can see Vita doing well for 1.5 to 2 years, and then it will begin to fall further and further behind the 3DS. I do not think it will do as well as the PSP simply because Sony Playstation is not nearly as big of a brand as it was when the PSP came out. The technical advantages of the Vita are not as great as PSP had in comparison to the DS. I will guess 3DS does 130 million sales, and Vita does 45 million by the end of 2016.