The dedicated handheld market isn't dead, it has decreased for a few factors.
The 3DS/Vita don't offer anything a smart device can't + to a casual they just look like upgrades, not the next "big" thing. What sold the DS was the innovation, and just strange concept of a 2 screened game system. The PSP at its release looked amazing graphically and had tons of features that most phones wouldnt be able to do for years.
The 3DS isn't innovative because it hasn't shown the public anything new like the DS did at the time. The Vita still looks nice, but people aren't willing to pay that price for a handheld and the memory card prices absolutely kill it (its internet browser isn't that great either, plus lack of apps make it unappealing to the general public).
I believe that people have less of a reason to own a dedicated handheld device at the moment, because while the software may be good if the price of what is required to play it isnt justified, no one will buy it. I'm very confident that Nintendo will be making another handheld (not too sure about Sony), but if they continue the DS/PSP line, they will receive less sales because the market is tired of those lines :P
