Because they simply are.
They are less powerful then the home consoles,no huge 3A games for them and most people see them as toys.
I do most of my gaming at home so I'd rather play a decent game on my home console.
Even if I traveled alot I'd still wouldn't be using handhelds,I'd rather watch a show/movie on my phone.
And they are unpractical,especially let's say psp or a vita.You can't even get a movie to work hassle free on them,and I'm not that hardcore that I have to spend every free minute of my time playing a game.