It's easy to say they didn't in retrospect because of the success of the Switch, but the only consistent success Nintendo ever had was with their handhelds.
They stopped having success with their home consoles because of the lack of third-party support. While Nintendo's first-party titles have always dominated the best-sellers charts on their systems, there was a time when third parties rose to prominence and did well on their consoles. The NES & SNES both had many hit titles from companies like Capcom, Konami, Square, and Enix. But those major third parties effectively bugged out and moved to PlayStation in Gen 5, largely because of the decision to stick with cartridges. While the N64 did have the support of several Western companies, with a very successful partnership with Rare as well as several modest hits from Acclaim (Turok) and THQ (WWF & WCW games), it barely had any support from the major Japanese third parties. The loss of Final Fantasy and Dragon Quest in particular was devastating (the latter more so in Japan). FF7 is notable in that its release marked the point where the PS1 really took off.
If the N64 retained the level of third-party support the NES & SNES had and kept those big titles, it likely would have easily won its generation seeing as the PS1 started off slow and the N64 was, at least in the U.S., strong out of the gates. PlayStation may have been a strong competitor, but Nintendo would likely have continued having very strong home console sales. Imagine if the N64 had FF7, Resident Evil (the belated, downgraded port of RE2 notwithstanding), Metal Gear Solid, Dragon Quest, Tomb Raider, and so on. Imagine if the N64's successor had freakin' GTA.
Without the full backing of major third parties, Nintendo's home consoles struggled, with the only hit they had after the SNES being the Wii. With only one hit out of four generations, it was pretty clear that Nintendo needed a change of plans. Their home consoles couldn't consistently do well, but their handhelds did, so they made the Switch a best-of-both-worlds machine: a powerful handheld designed to also function as a home console. Putting their eggs in one basket was risky, but it turned out to be the right decision. Had Nintendo done more to keep strong third-party support on their home consoles, the trajectory of that market would have been very different, and we probably wouldn't have ever had the Wii, Wii U, and Switch. We'd probably be on, well, whatever they would have named a hypothetical Gen 9 console that competed directly with the PS5.