No one's buying a Nintendo phone, the OS wouldn't be able to measure up with what people expect. A shoddy OS on a console is OK because it's not a central part of people's lives, but their smartphones are basically their all purpose hub for their business/social life, people have a total freak out at Apple when one of their apps (the Maps one) doesn't work properly for a couple of weeks. Adults would lose their minds if their smart phone had even 1/4 of the issues the Wii U OS does.
Beyond that, Apple would outmarket the living crap out of them if they felt they were any kind of serious threat. That's just not a fight Nintendo wants any part of I don't think.
I'd rather see Nintendo allign themselves with one of Sony or MS (pick the best offer) for a few years, making games for their home console, while getting their support for a future Nintendo handheld + a metric ton of cash/royalty kick back as part of the deal.