It's because they are the secondary devices for the companies. If you look at history for Sega, Sony and Nintendo, you will see that while the handhelds have some exclusive games, a great portion of their library is made of inferior versions of console franchises, spin-offs, prequels and so on. PSP received God of War prequels, Vita got an Uncharted prequel, 3DS gets inferior Mario Kart and Mario games.
The design of the machines itself shows that they won't push any boundary. Hell, we only got twin analog sticks THIS gen and only on Vita. This is standard on home consoles since the PS2. New 3DS brough 4 shoulder buttons, again: this is on home consoles since the PS1!!! Still waiting for triggers.
The way people see handhelds is only a reflex of how the gaming companies treated this devices. If Eurogamer is correct and NX is indeed an hybrid, then we will get the first handheld that will be a first class citizen in the gaming world instead of a spin-off/prequel/inferior version/rehash machine.
Don't get me wrong, I love portables. But if I had to chose between a home console and a handheld, home console is the way to go. I myself only got a handheld this gen and, if I had to chose only one, neither the 3DS or the Vita, or even both combined, could replace the PS4. Gaming only on a handheld means losing 80% of the good games.