Metroid Other M didn't sell because it completely goes against the grain of what the Wii console itself and thus its fans represent, which is less cutsenes/ cinematic values and more action/freedom in game to allow the gamer to explore and do what they want. It is arcade style values held primarilly by retro gamers/non gamers that makes up a large part of the Wii userbase.
They tried to make an HD style game; by this I mean a game that emulates the games on those consoles (not the graphics), with cutscenes, dialouge, character developement, story, tutorials, linearity, etc.. which would be bad enough, but they made it on the wrong console.
It also didn't sell because it is such a radical departure of the traditional Metroid games, which were generally much more successful. Metroid is about being in solitude and exploring vast dangerous areas with dangerous monsters. It is about gaining abilities throughout the game that allow you to progress and feel more empowered and fast paced action. Other M seemed to do everything in its power to divert from these principles of Metroid.
