Calling the N64 and Gamecube moderately successful is a pretty positive outlook considering the N64 floundered when compared to the PS1, and the Gamecube just barely, and I mean barely sold enough to turn a profit. That aside, yeah, Nintendo hurt themselves. With the Wii, they struck a goldmine, but that goldmine held a dark future. They found the casual userbase, a group of gamers that had yet to be heavily appealed to by any piece of gaming hardware. What happened then? Nintendo became the brand for the casual gamer. That worked well for one generation, but last generation was long. It gave Nintendo a defined image for a whole set of gamers who were growing up in that time period. Then the Wii U comes and Nintendo says "this will appeal to hardcore gamers!"
Yeah, Nintendo, I hate to say this, but you appealed to a very specific crowd who has come to enjoy smartphones and tablets now. Nintendo effectively lost at least 85% of the Wii userbase to more casual friendly hardware that tons of people own now anyways, making it a convenient platform for casual gaming. Literally all they have left is the most dedicated of Nintendo fans.
At the same time, you have to consider that Nintendo has been struggling since losing third party support in the N64 era and failing to regain it in the time of the Gamecube. They created a very complex situation for themselves after the Wii that makes success in the home console market something that will be nearly impossible for Nintendo as time keeps moving. There are no more hidden audiences to appeal to. There are no more audience based flukes that will lead to huge profits.
