Outside of a comic book convention, I've still never seen a grown adult (or even an older teenager) with a Vita or 3DS out in public in the US/Canada. And just in the last month I've been to Seattle, Vancouver, Las Vegas, and Los Angeles.
Smartphones have totally killed that part of the market, Sony/Nintendo need to hope that kids still want handhelds enough, but even that market is under assault by cheap mini-tablets.
Ancedontally too, I had many friends in their 20s who got a PSP near launch when it was the "hot" new thing for 2005 and I knew of others who had a DS especially after the Brain Training craze hit.
But none of them have gone back to buy a Vita or 3DS.
In Japan handhelds reign supreme simply because they've effectively replaced home consoles.
