I think the biggest issue with the Vita is that it didn't have many of the big games.
When you look at the best selling PSP games, there are Grand Theft Auto, Gran Turismo entries. A lot of these were some of the biggest franchises at the time, and they had (at least timed) exclusive PSP versions.
Most of these franchises were outright missing on Vita. Vita's biggest representation was the worst rated CoD game of all time (and Minecraft at this point).
I don't think it had anywhere near enough support to sustain a console.
I would say though that expectations will be different. The Vita had to sell enough to be worth making games for. If the next one plays PS5/PS6 games, then it doesn't even need to sell as much. There's no risk for making a big game for it.
It largely just needs to beat out the R&D, manufacturing, etc of the hardware itself. Whereas before, Sony would need to be cognizant that they were losing however many millions on the games too.