Nintendo's biggest mistakes, in reverse order (biggest last):

4. Forcing developers and publishers into highly restrictive contracts, and then screwing up their deal with Sony. This gave Sony the opening it needed to become a haven for devs and pubs, and thus resulted in a massive loss of 3rd party development to Sony. Sega had the same problem, with the same results, but didn't have a strong enough first-party lineup to withstand it.

3. Allowing Rare to die, and then selling it. If Nintendo had acted earlier, they could have instead bought Rare before it started to die, and given it what it needed to grow and flourish further. Rare's talents were on show with some of the greatest games of the 4th and 5th generations, including some in genres that Nintendo still lacks experience with. Rare would have filled that void, and Nintendo likely would have had a huge FPS franchise by the end of the GC's life. This would have set Nintendo up for the 7th generation in a way that would have made the Wii dwarf all other platforms.

2. Failure to provide a user-friendly and dev-friendly online system for a generation and a half. The issue isn't online competition with MS or Sony, but developer support for the systems. It's more likely that Modern Warfare, for instance, would have been on the Wii if it had a robust online system with easy friend code sharing (Nintendo never actually had to get rid of friend codes, just make them more user-friendly).

1. The biggest mistake of all, was failure to make real use of MotionPlus. It isn't just a coincidence that Wii sales began to struggle about 6 months after the release of Wii Sports Resort - Nintendo barely made use of the updated technology, and didn't even make extensive use of motion controls. With third parties not supporting the Wii well enough, Nintendo should have been stepping up to the plate with many top-notch games. It seems that they were already shifting focus over to the 3DS by then, sadly (not that the 3DS wasn't worth focus, but it could have been delayed an extra 6 months without real impact, or an extra 18 months with relatively minimal impact). If not for Nintendo's failure to truly capitalise on the Wii and MotionPlus, Nintendo would likely still be dominating the hardware charts with the Wii today, and Nintendo could have put off the Wii U a little longer, giving the chance for better hardware at a lower cost, and allowing them to spend more time and money on convincing third parties to support the system.

That biggest mistake is the biggest because it's not so much about Nintendo's past, as it is about Nintendo's future. They would have been in a much stronger position for Generation 8, and could have asserted even greater dominance. Now, they're going to have to fight again, just as they had to with the Wii, to get third parties to take them seriously.

Idk about virtualboy.
But I'm prety sure the worst is 3ds price cut.