The UMD didn't fail because people didn't wanna watch movies on PSP. It failed because of high prices, little features and low selection. I know friends who downloaded dozens of movies and watched them on the PSP. I always saw people watching movies on their PSP's, just rarely using UMD's.
What is Nintendo going to do differently? First off you will have one option when wanting to watch a 3D movie on 3DS. That option is to buy the cartridge. You won't be able to download every movie you want for free. Also consumers who owned PSP's could watch movies on their portable DVD players, Phones and laptop's. But their will be only one 3D portable device to watch on meaning higher sales. Also don't forget all the people wanting to watch 3D movies but can't afford a 3D TV. I couldn't even afford a 32-inch, but I'll buy a 3DS right aways.
People wanting to watch 3D movies would be pushed towards 3DS. Also UMD's were very low quality, I still can't hear the audio on most UMD's. The picture is bad and their is no special features or other features. But if Nintendo did this right and so did Hollywood 3DS movies could be of optimum quality, a movie theater on the go.
I think if Nintendo allows 3D content like Avatar and Resident Evil Afterlife and not just childrens movies then the 3DS could do what GBA video and UMD failed. I don't think 3DS sales would rival Bluray and DVD sales but I bet 100k could be sold (A new title) easily if the title was big enough.
Now if your able to download 3D movies and watch them illegally on your 3DS then sales will flop. But if Nintendo protects the console from piracy then I think 3DS being the only handheld 3D device will sell butt loads.
Not to mention a portable 3D player would sell to non-gamers. Their are no other portable 3D players they could fly off the shelves like portable DVD players did. Imagine if Nintendo released a big screen version (Like DSi:XL) consumers would go mad. There is a market and Nintendo could tap into it.