The problem for both handhelds is the iPhone 5 and iPad mini are set to release this September apparently.
I think Apple is going to cannibalize a lot of people's holiday wallets this year, especially for portable gadget spending.
The Wii U I think could hurt both handhelds to an extent too. For the people who like handhelds at home because of the quick start up, ability to play away from the TV/in bed/whatever ... Wii U can fill that gap with real console quality graphics.
I have both a 3DS and Vita, but honestly I play them mostly around the house. When I'm out of the house, I just don't have the extended free time to really sit down and get into a game. But with Wii U, y'know, I think that's going to eat up a lot of that handheld-at-home play (real AC3 > spinoff AC).