Retro Studios can handle any genre Nintendo wants. It's just a matter of giving them the budget to do so and maybe hiring or insourcing 20-30 extra staff which is no problem for a company.
Nintendo is BIGGER than Activision or EA, lets not forget that.
That's not the issue for Nintendo. They issue is simply they don't want to make those types of games.