Suppose Nintendo had beefed up their specs to the point of the X-Box 720 and PS4. How much do you suppose that would cost them? Let's say... 70 dollars. Would that be a fair estimate? First parties make about 7 dollars per third party game sold. So, Nintendo would have to sell 10 third party games to make up the added cost. The attach rate for a system is, on average, about 10. So... the odds of Nintendo winning out on this would be slim. Of course this is an oversimplification as other factors come into play, but financially a more powerful system would not likely work.
The other option would be eliminating the tablet and using that money to beef up specs. If they did that, we'd have a Wii U that would be an X-Box 720/PS4 but with the added benefit of Nintendo first party software. THIS would be the equivalent of a Gamecube, and that strategy would work out poorly. If Nintendo aimed at the same market as Microsoft and Sony do, they'd lose. Unless Nintendo radically changed their development strategy and alienated their fanbase AND did a 180 in terms of marketing, they'd struggle very mightily to tear Call of Duty fans away from their console of choice.
So, Nintendo did what they do. They focussed on new control gimmicks (gimmick as in: an ingenious or novel device, scheme, or stratagem, especially one designed to attract attention or increase appeal) and kept their marketing broad rather than trying to zero in on the particular sect of 16-34 year old male gamers that their rivals appeal to. The strategy hasn't panned out perfectly so far, but I suspect that once Nintendo rolls out some software that better uses the Gamepad, things will turn around. It is a very cool idea that Nintendo has done a poor job at marketing (why Nintendo chose to put Rayman Legends in their demo units instead of NIntendo Land stuff is beyond me).
As for how to appeal to that "hardcore" demographic, we've seen Nintendo's strategy with Bayonetta 2. Nintendo's idea is to use their money to fund software instead of beef up hardware. Is Bayonetta 2 going to make a gamers choose a Wii U over a PS4? Nope. But, it will go a long way towards making the Wii U worthwhile as a secondary console. I suspect we're going to see a lot more of Nintendo acting as a publisher as time goes on.
TL:DR version- Appealing to the "hardcore" demographic is not as simple as shoving more RAM into a box. Making a system that would compete with PS4/720 on the hardcore front would be a losing strategy. Nintendo's current strategy has not paid off just yet, but its chances of success are far greater than the suicidal approach people seem to be advocating.