My thoughts on Madworld, Zack & Wiki, etc.
First off, one of the main reasons these games don't sell is because they don't cater to the majority of Wii owners. People who bought the system to play together (Wii Sports...). These are all single player games. People who want single player games are better served with HD consoles. People buy Wii for multiplayer 'family' fun. Although it's a shame that so few of those 6.5m Zelda owners bought Okami.
Secondly, while people grew up with Mario & Sonic so they appeal to all ages, if you otherwise put cartoony characters in a game but give it complicated gameplay only adults would love it's a recipe for disaster. This has been done time and again on Wii from Opoona to Little King's Story. It looks like a good game for a child but it isn't, nor does it look like something Dad would want to play. Actually, Mario & Sonic almost never have complicated gameplay anyway.
Third, whatever made Sega think adults want M games with perfuse blood and more f-bombs than a biker fight is beyond me. Mature people generally want a deep and compelling story to go along with a bit of blood and appropriate swearing (and nudity). Again, a M game that Mom and Dad could play together and even in front of the kids, would sell a lot better.
Madworld's odd art style and over the top vulgarity and violence definitely made it niche, especially on Wii. But that's okay, there's a market for it. However, it was short and repetitive too. GREAT rental. Not really worth a purchase.
I think Suda51 is doing a great thing with No More Heroes 2 making the action and mini-games and such more arcady, more accessible and generally more fun. It'll still be lucky to make 500k but should sell even better than the original.
The other problem is the general public doesn't buy often so they stick to what they trust, which makes branding very important. Nintendo's brands (Mario, Zelda, Wii, Kirby, DK, etc) are well known. Licensed brands do well on Wii from Lego SW to Hanna Montana to Monopoly (which are all multiplayer too btw). These quality games have no such recgonition making sales even more difficult.