There is a problem with your theory. Before the N64, the main competitors were Sega and other smaller companies. Since the N64, the main competitor has been Sony (and after the GC, Microsoft). These are two of the biggest corporations on the planet on hardware and software respectively. Compare them to Nintendo, that focuses just on making videogames and consoles for them. Competing against them by doing the same as them won't be succesfull because Nintendo doesn't have their resources. Their main strenght is their IPs and their ability to reinvent themselves.
Also, I wouldn't call the Wii a failure. You say the Wii alienated costumers, but if Nintendo had tried a GameCube HD instead of the Wii, Nintendo would be in a worst situation that they are now finantially.