They could probably take Microsoft if they did everything right and got some better management in place, but the problem is they'd be forced to engage Sony head on whether they want to or not in that case and MS would also force them to spend money beyond what they want.
I think after the early 2000s and Yamauchi realized that Microsoft wasn't going to fail massively in the game business, and seeing how MS just casually threw several billion in his face to buy Nintendo entirely ... Yamauchi changed and so did Nintendo.
From that point on Yamauchi instructed Iwata and all his staff that they had to be different and could not compete head on as it would be suicide financially as MS could outspend Nintendo no matter what. Before that time though Yamauchi had no problem with direct competetion and was often a vicious competitor.
Microsoft is what changed Nintendo I think, they got scared at where the industry was going and didn't want any part of that, it's like being in a poker game and seeing the money on the table getting too rich for your blood. Time to bail.
They couldnt even beat MS when they just started, it would be much harder to do now