These I find inarguable:
3D and 2D Mario
3D and 2D Zelda
Splatoon (now)
Pokemon
Mario Kart
Super Smash Bros.
Animal Crossing
Luigi's Mansion (Dark Moon sold over 5mil, if they make another one, it'll be a big deal)
Donkey Kong
So close, but not quite:
Kirby
Fire Emblem
Other interesting cases
Brain Age used to be, the 3DS entries underperformed
Nintendogs, see above
Metroid and Star Fox have passionate fanbases, and I feel like they could be AAA, but need better treatment from Nintendo, which Metroid seems to be getting.
EDIT: Xenoblade is another interesting case as they all always have the massive scope traditionally associated with many AAA games, but I feel lack the budget, marketing, and/or sales potential to be AAA. Will be interesting to see how Xenoblade 2 pans out. ARMS could be if Nintendo supports it like Splatoon.