Everybody knows that excluding Wii, Nintendo consoles have declined. The general thought is this is due to the added competition from the likes of Sega, Sony & Microsoft coming and stealing their audience. Well I have another theory, Nintendo wasn't losing their audience to the competitors, their audience was gradually shifting from Nintendo consoles to Nintendo handhelds. Let's take a look at Nintendo hardware sales per generation pre-Wii.

NES-62 million, Game & Watch-43 million

SNES-49 million, Gameboy (89-96)-54 million

Nintendo 64-33 million, Gameboy (97-03)-64 million

Gamecube-22 million, Gameboy Advance-81 million

Each generation sold 100 million give or take a few million, with the handheld sales increasing at roughly the same rate as console sales declining. So what exactly would be the reason for people shifting from from one to the other?

Nintendo's best selling games and consoles are ones that emphasize on simple, pick up and play, casual experiences. As consoles became more complex by adding more face buttons, shoulder buttons, triggers, analog sticks, the games became more complex by default. Nintendo handhelds on the other hand retained their simple, casual style thus consoles began to decrease and handhelds increase.

With Wii & DS, Nintendo created the ultimate casual experiences thus sales exploded, with Wii U & 3DS they have reverted back to making more complex control schemes and games thus it is on track to becoming the worst selling generation in Nintendo history.

Pretty much proof that Nintendo will go back to a more casual approach next gen...

Yep and that's best for Nintendo and consumers

Thats debatable, but whatever...

Not really. Nintendo said that the casual market was fickle and lets not pretend that the Wii U wasn't supposed to attract the casual market aswell. 

problem is  you need a gimmick to reach a new market, if Nintendo find one strong enough to base their console on then great, if not then they have go for the core market where theres tons for them to gain if they play their cards right. Alternatively they accept Wii U level sales and just focus on profitability of hardware at launch

Well like I said, Nintendo's casual games and consoles are much more popular than their "hardcore" games and consoles so it's not really debatable which strategy is better for Nintendo.

As for consumers, yes it's debatable, but the way I see it, what benefit is it to have 3 nearly identical machines on the market at once? An increase in options and variety is always a good thing.

The WiiU is not like the other 2 at all...

How? One could argue that both Wii U with the tablet and 3DS with 3D were pretty much directly targeted at the casuals.

If there is one thing you can't accuse Nintendo of this gen it's that they pandered to the core gamers too much.

It's true that the handhelds have always had lesser buttons, but even handheld games, like Pokémon, are complex. More complex than a SMB 1 game, for example.

I'm not really sure that the decline of Nintendo's home consoles can be explained just by referring to complexity in terms of button layout or in game complexity.
Otherwise, how can you explain the success Sony has been having with PS? Both it's controllers and games have always been complex.

The best way to understand the decline and success of home consoles and handhelds, respectively, is to look at each other separately.

Handhelds were always the cheaper alternative for a lot of parents. Not to mention that they were portable.
Home consoles by Nintendo started suffering because of competition. And we are not talking about competition in the same market...
Since SEGA, the gaming market has shifted away from Nintendo's type of games to, let's say, Sony type of games.
The difference between Nintendo and Sony's focus (meaning, the type of gamer) - and strategy- had enormous impact on how the market would (re)act.

Sony's approach to the market, won them the market pretty much since the beggining.
By focusing on the teen, late teen, young adult segment, they managed to succeed the same way Nintendo succeeded when entering the market: they offered something that the market wanted but didn't have.
Problem was that Nintendo didn't evolve with the market and got left behind.

You could almost say that Sony ripped what Nintendo saw (spelling?).

Wii and DS were major hits because it introduced not just HW innovations unlike anything seen before, but because they also went after a market.... no, they created a market, by offering an accessible and inviting HW (Wii remote and touch screen) and games that complemented that same HW.
In this case, simplicity was key!

You can see it in the games they make. MK8 and SSB4 are more aimed at fans of the franchise than the casual crowd. Games like Bayonetta 2, The Wonderful 101 and DKC: TF are games for core gamers. Wii Sports and WiiFit got way less attention from Nintendo this gen. Pikmin 3 and Star Fox are also proof that they wanted to cater more to the core gamers.