They need another Wii basically. Something unique that generates a great deal of discussion the moment people see it. But they can't simply repackage the original Wii, that is old hat now, they need something new.
Unfortunately making a new idea like that is very hard, and making it affordable can also be a major problem.
Even the handheld/portable ... I think they may have to push for a new type of form factor or something really different about from previous handhelds so that it can be unique in the smartphone/tablet world.
Unfortunately I don't think Nintendo can realistically compete with Sony/MS. Even if they made a comparable/better system, the abundance of kid-friendly mascot games on the platform would get it labelled as the "Playstation for kids", and even kids don't want that.
The more different/wacky the NX is in an immediately understandable way, probably the better for Nintendo.