While I think the High Price and lack of games effected it. I think perception was made worse by the extra charges for the memory card - when you nickel and dime consumers with extra little fees, it's a big turn off. It makes anything seem lacking value.
Mostly I think it's a dying market. Kids still want and will get a 3DS or a 2DS, but why carry around a Vita when your phone can do more? The Vita is more expensive and the market it should appeal to has a phone. True a phone is not as good as a gaming device, but that's what consoles are for on a nice big screen.