They need to attract either the hardcore non-Ninty gamer or the casual, preferably both. For the former, they need 3rd party support (meaning easy to dev for / port to), the latter means having a certain something to capture the imagination of non-gamers. (That 'something' doesn't necessarily need to be a gimmick. It could be a new game that does interesting things.)
If they just want to be successful they could go 3rd party on consoles and keep doing handhelds. But at this point I think Nintendo measures success by staying in hardware, not necessarily profit.