Well, they're trying a bit with W101 and Bayonetta, by using third-party developers. W101 looks like a bomb but we'll see how Bayonetta 2 performs.
If both of those fail then I think we're going to have to look at the possibility that the current Nintendo fanbase isn't going to care much about games that don't use one of Nintendo's established characters. On the other side of the equation, Sony and Microsoft have so much content, both their own and from third-parties, across so many different genres, that Nintendo isn't going to easily be able to compete in those areas just by creating one or two new franchises. If they made a new FPS, for example, it would still pale in comparison to what the other systems have in that category.
Honestly, Nintendo should probably continue being mostly a Mario/platformer box and just live with the realization that the Wii was an anomaly. If they sell hardware at a profit then there is nothing wrong with being a niche player. The truth is, I think, that Nintendo can either produce low-cost, low-spec hardware OR they can have the same third-party software as Sony and Microsoft. Not both. If they decide to continue down the first path then their fans are just going to have live with it.
