1. The format. Whereas the CD was mostly a benefit for PS1 (minus load times and scratches), UMD was an anchor for PSP. DS wasn't really trying to compete with PSP in specs like N64 was with PS1, so the much smaller storage space of the DS Game Cards isn't a big deal like it was on N64. Sony realized this eventually and released PSP Go as well as using Game Cards for PS Vita. Optical drives on a handheld are not a good idea.
2. PlayStation is simply better at making home consoles than handhelds. Just because they kicked Nintendo's butt with PS1 (and especially PS2) doesn't mean that will translate to portable consoles.
3. DS had first-party games that lit the world on fire. Nintendogs, Pokemon, Mario, etc. Also, in terms of the library DS and DS Lite can play GBA games.

You mean like the PSP dock?

Third party support.
/topic

Because Sony misread the handheld market pretty badly. Eventually they did get smart, at least in Japan. They thought everybody wanted a mini PS2.

Western gamers wanted stuff like Nintendogs and Pokemon, not stuff like portable Uncharted or portable God of War. Japanese gamers just wanted more RPGs. 

It looked like it was going to be the case until the DS Lite launched.

But the DS was also released at the same time and had backwards compatibility too. So not entirely comparable to the N64.

Not to mention that around the time the DS came out, dedicated compression codecs and APIs were becoming cheaper and more readily available to developers. And with DS game cards having much larger capacities than N64 cartridges, it was much easier for developers to incorporate FMV, streamed audio, and voice work into DS games (abiet at a lower quality than PSP) than it was for N64 games.

So the storage advantages that UMD had, ultimately never mattered that much in the end.

Another thing was that PSP lacked killer apps compared to the DS.
Outside of Monster Hunter in Japan and maybe GTA in the West, it had no answer to DS megatons like NSMB, Brain Age, Nintendogs, Mario Kart, Pokémon, Animal Crossing, etc.

Only 1 PSP game sold over 5 million copies, while the DS had 15 over this threshold.

Yeah, pretty much. All of the features that gave the PS1 the edge against the N64 were actually drawbacks in the handheld market.

Two big factors:

1. Pokemon
2. Price