Part of the reason in the past was that Miyamoto himself doesn't believe in catering to a demographic where he, as a responsible parent and son of strict teacher, couldn't let his kids play.

Others have also pointed out that Nintendo doesn't really care about aethetics in most of their releases, until after they get a game play structure set. Once that is done, it usually falls to the higher ups, Miyamoto being one of them and they determine how the game is marketed.

And lastly, Nintendo's core groups are all stuck on legacy duty. The team that worked on Mario Galaxy has jumped from its sequel to 3D Land. Zelda's core team is locked into working on ways to make the next Zelda better. You have a team for Mario Kart, you have a team for Mario Party/Wii Party/Sports/NintendoLand mini games, etc.

So instead of having their core teams work or waste time on games that upper management won't approve, any thing that doesn't have a dedicated team gets shuffled to a second party developer or a third party that is willing to do the work.

Which is actually how a lot of Japanese companies are handling their games these days, outsourcing to other development studios. When you see a Mario Kart or a core Mario platformer game made completely outside of Nintendo's core studios that is a sign the Japanese gaming industry is in horrific trouble.

Also most third parties really don't care, they'd just see it as more reason to stay away if Nintendo made a really good game in a area they were thinking about making a game in because they know they wouldn't compare.