If you are really into the complex and demanding, esp fast paced, twitch, games then touch screens simply don't cut it. I think phones are a gateway device for gaming. Lots of people will be satisfied with the limited gaming capability that touch screens can provide, but many people will not be satisfied and will want the greater sophistication, depth, range and subtlety that buttons and analog sticks can provide.
It does mean, however, that there is little incentive for casual bejeweled, temple run etc type gamers to buy a handheld console ever. If all they want is something to amuse themselves on the subway or in the loo then all they need, and all they'll get, is a phone, Apple or Android doesn't matter. And that's why any announcement of a new iPhone is largely irrelevant to gaming because there is far more to smartphone's effect on the casual end of gaming than the annual Apple upgrade.
So the handheld market is smaller now han it was, but it will be pretty stable long term, because there will always be enough people wanting what handheld consoles can only deliver.
PS Now on the Z3, with DS4 (or purpose built attachment that is less cumbersome than DS4), I think is more of a threat to dedicated handheld consoles than any iPhone. Esp if that gets extended to any sufficiently powerful Android phone (Galaxy S6/7, HTC, LG). Sony won't care as long as they get their PS subs the hardware on which the games are played won't matter to them. I could see them making a Vita "skeleton" that allows different sized phones to be slotted in to where the Vita screen would normally be.
“The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt.” - Bertrand Russell
"When the power of love overcomes the love of power, the world will know peace."
Jimi Hendrix







