Currently Online 
If we look at the last two Nintendo console's that had any real marketplace success we find the Wii and N64 (particularily in North America early on sold huge numbers).
But if look at what caused the success of those two consoles, we find breakout hits bringing in a wider audience than Nintendo would have otherwise:
GoldenEye brought in lots of non-Mario fans and kept the system afloat for pretty much like a year until Nintendo could get Zelda:OoT finished.
On Wii ...
Wii Sports was actually largely driven by college age-adult age gamers.
Both games are successfully in part because they are not so overly cartoony/over-mascot-ized to turn off adults. Wii Sports would not have the same appeal if it was "Mario Sports" instead.
And GoldenEye would not have the same appeal if it was "Mario Paintball Shooter" either.
Games like Bayonetta, Devil's Third, Xenoblade, are too hardcore/niche and don't invoke what people really like about Nintendo games (easy to pick up and play, multiplayer oriented), but people don't want mascots in every game and this is a problem for Nintendo. So they are not the answer either. Even Nintendo Land got this wrong ... went too cartoony/child-like in presentation.
Mario is like a nice colonge, it's fine to use it, but if you douse yourself in a colonge, even one that smells nice, it's going to be a turn off to people.
Nintendo needs to find new IP that connect/resonate with adult/teenage gamers in the way GoldenEye and Wii Sports did (two very different games). Until they do that, they will be stuck selling to basically the same crowd of "Nintendo Lifers", which is only 10-15 million strong and a small portion of the kids market (kids like any type of video game, so Playstation or XBox is just as appealling to them as Nintendo is). If that's your audience, you simply will not have a successful home console.
