It's not a single factor at play here. The "hidden" cost of the proprietary memory cards doesn't help matters considering most tablets and phones already have a decent amount of storage available internally or are bundled with a substantial sized non-proprietary memory card.
Sony don't seem to know how to market the machine either. They seem to be trying to sell it as a media machine or a mini tablet. It's understandable as stand-alone portable games machines are now relatively niche unless they truly offer something different but people have smart phones and tablets now that do a lot of what Vita does. They're failing to communicate the unique aspects of the system or those aspects aren't enough for people to shell out the money.
The games are probably the best aspect as the Vita has a nice library for a relatively young hand-held and the price for the hardware is actually very good. However, other than a small and core niche, consumers are not seeing the value in the machine at its current price point. If the new Google Nexus tablets are anything to go by, lowering the price should be possible in the not to distant future.