Or more simply in picture form, this happened:

The dedicated game handheld is just another casualty of the smartphone boom, much like point and shoot cameras. 

The proposition of what a game handheld is is pretty silly to be honest in this day and age -- Sony/Nintendo want you to pay them $170-$200 for a large brick that primarily just plays games, at $30-$40 a pop. What older person even has the free time to play all the home console games and then needs to play like another 3-4 hours a day on the road? 

The main audience left for gaming handhelds are kids who are too young to have a smartphone. But Nintendo dominates the kids handheld market, so Sony has no "in" there. Even that section of the market is shrinking as a lot of kids are opting just to play on their mom/dad's iPad. Nintendo is feeling the hurt too. 

Vita didn't catch on because it had no must-have games for the casual gamer, no marketing, and Sony's pretending it didn't exist.

Even the PSP to be honest, once the hype of the first few years wore off where it was supposed to be the next "iPod" , its sales were mediocre in North America and in Japan it was saved by the phenom of Monster Hunter which ensured a steady stream of Japanese support.

The decline was already on before the Vita even launched. Selling a game handheld to the audience Sony specializes in -- the 16-35 year old male is a dead end. They don't want a game handheld, their smartphone and their wallet/keys are the only thing they want in their pocket.

While I agree with all points on here as to why it failed (mostly games)
It also had so many features going for it. numerous touch controls, remote play.
But I think it was a culmination of things I just pointed out the design, and power aspect. (It was a gripe for me)

Sony and those memory cards are a big deal breaker too, while I understand the reason for it, It would have been nice for them to sell at a much lower cost especially considering since the psp go they have been pushing digital downloads.

On a side note, Both the Vita, and 3ds have seen reduced sales from previous hardware. Does anybody else think this was due more to them being so hard to hack for piracy? I know this is a taboo subject on here to talk about. but if a vita exploit did show up, how big of a hardware boost do you see it getting?

I still fail to see the need for people to speak for others. Please feel free to speak for yourself but not others. At least say "most" if you absolutely have to. Your statement as phrased is actually ignorant and offensive.

It needed more games and memory cards were too expensive. But the biggest reason is that smartphones have taken away many potential customers and so the market size has fallen significantly.

Memory card was the main reason it took me a long time before I purchase a Vita.  I loved my PSP but when the vita came out with those very expensive memory cards, I was like "I can wait until the price drops".  I never ended up buying one at retail but instead a person I knew was unloading one for a very cheap price.  As far as handhelds go I think its definitely one of the best.  

In the end, it was the combination of price, memory cards, marketing, support and competition from other devices like smartphones.

Smartphone shrinking the market+no mass appeal games(Sony should have moneyhatted exclusive GTA) and high price. PS4's success as well, now they have a winner in their hands they can write off the Vita

Most of the reasons brought up here are far from the case (Imo). It's mostly because of marketing and to a less extent due to lack of games. But even if it were to get many great exclusives, there is no guarantee it would sell well. Never had a Saturn or Gamecube but from what I've heard from fans of those consoles, they had great games but still didn't sell well.

Since we're talking about masses, I failt to see how the points you brought up would make a difference.

- Size: It doesn't fit in your pocket, but neither did the psp, nds, 3ds and they were/are all successful. Who has space in their pockets for a wallet, a phone and a handheld? I always put it in my bag.

- Battery: how much better was the battery of psp? How much better is a mobile phone's battery? People learn to live with it.

-Lack of exclusives- see gamecube, wiiu (it has way more AAA games including huge franchises and still struggles). How much did Killzone or Tearway help when they were released? What about LBP? Yes, more would have helped, but still not enough.

- Memory cards: most casual gamers who buy one know close to nothing about memory cards. People were fine buying one for ps2 games and ps2 sold pretty darn well despite the extra cost of memory cards. It's about perceived value. 

Here are some other possible reasons you should think about:

- How many people had a smartphone when the psp launched? It was a novelty. Sure games like GTA helped, but at its time it was a unique product. Now all the extra feautures that it has can be found on a tablet/smartphone. How well are handhelds doing at the moment? A couple of years ago nds and psp were leading sales and easily beating every other console. Nowadays, even the 3ds is struggling. It has nowhere near the same success especially in the software department.

- 3ds price cut. People interested in a vita expected a price cut because the 3ds received one a few months after it was released. While waiting for one those people lost interest in the console.

- Doom. As soon as a console is not perceived as a cool item to have, there's pretty much no going back no matter what a manufacturer does. This happened with the wiiu. As soon as it stopped selling amazingly well, it was doom article after doom article. Sort of same thing is happening with xb1 hence MS's many efforts to help the console with any kind of deals (kinectless xb1, price cuts, free games, 3rd party games, etc). Sony and especially the playstation divsion has been in a terrible state because of the ps3 and they couldn't afford what MS is doing at the moment or what Ninty did with 3ds. 

-Psp owners getting burned. Same with wii and wiiu. The last couple of years of psp were terrible because of piracy and left a bad taste in people's mouth. They didn't feel like upgrading to the next console in fear that piracy would ruin their console again. The ps3 on the other hand had a great couple of years and the ps4 is doing well. Customers were happy and upgraded. 

Of course, I could be wrong. I'm basing my observations on comments from friends who owned psp's and didn't get a vita or the ones who did buy a vita and told me what they knew about it. My point is: the majority of people who buy consoles are not hardcore, well-informed gamers, but people who buy a console because they heard great  things about it. Marketing helps more than people realize.

Edit: I fogot about Monster Hunter 4. I think that was what doomed the console the most. Japanese developers had no faith in it (imo) after that announcement and not only cancelled games for it, but big companies like SE were too afraid to support it with titles like Kingdom Hearts and Final Fantasy that helped the psp a lot not only in Japan but aslo in the west. I remember thread after thread on how the vita lost the exlusive that saved (/helped tremendously) the psp in Japan. 

The PS Vita didn't catch on due to several factors:

1. Pricey handheld...$200 is too much for most people.
2. Proprietary memory, and pricey...32 GB for $99?
3. Lack of games.
4. The PS Vita was 1" more then the PSP...why spend the extra $100?
5. Not softmod friendly. Sony has been pretty relentless on the PS Vita Security...making it less desirable.