Nope, lack of huge games did. It's not that people don't want to play "console-like" games, it's more that the games weren't popular enough. Are Mario Kart and SM3DLand not console like experiences, for example? Are they not very very similar to Wii/gamecube games?
The PSP had all the most popular PS2 games, except for Gran Turismo which came out late in the lifecycle of the PSP. GTA, NfS, FIFA, Pro Evo, Madden, Final Fantasy VII: Crisis Core, virtually everything was coming. The huge games were there, they just didn't become as huge as they were on the PS2. Why didn't they sell better? Because they were inferior versions on a more expensive machine for the most part. The PS Vita is in the same boat. People are better off going with a 360 or PS3 to play the games.
Mario Kart is rooted in the days when home console gaming was still very similar to gaming in arcades. Easy to pick up and play, highly addictive. That's the kind of home console games that work on handhelds. SM3DL is closer to modern home console games, but it has been adjusted to fit on a handheld. Levels are linear and don't leave room for exploration which would only slow the game down. It's not going to reach the numbers of a Super Mario Bros. game, but it has realistic chances to become the bestselling 3D Mario title of all time (which would be 12+ million copies).
Pay attention to this important difference. Not all home console games are the same. The question to ask for any particular handheld game is: "Would this game have worked in an arcade environment?" If the answer is no, then there's no reason to expect sales anywhere close to those of the home console counterpart. Keep in mind that this only works for the Western markets, because unlike in Japan, handheld gaming is perceived as second rate gaming over here.