3DS was a disaster in this area. I think GBA and Wii. Both fully supported their predecessor while being very successful launches.

GBA to DS

A fantastic machine with great library and big success replaced by a fantastic machine with even greater library and greater success

Certainly not Nintendo's strong suit, they need to make sure the next transition is strong.

NES to SNES: NES still got tons of first and third party games and the SNES came out of the gate strong.
GB to GBC: Similar to the NES/SNES
GBC to GBA: Similar to the NES/SNES
GBA to DS: DS was touted as a third pillar, but once it was successful it was deemed a clear successor to the GBA. The GBA still got a lot of support until 2006.
Honorable mention: GCN to Wii. The GameCube didn't have much after 2005, but still had some games in 2006 and a small list of multiplats in 2007. Fans still got Twilight Princess on GameCube, which is a good thing.

Tbh I feel like more times than not Nintendo hasn't had a great track record with console transitions. A big problem they'd typically have is taking too long to release their consoles, mostly cause the software teams weren't prepared to transition quick enough to the next console.
Imo I think Nintendo took a bit too long to transition from the NES to SNES cause it gave Sega a critical 2 year headstart where they were the by far the most capable system and allowed them to eat away Nintendo's marketshare early which set them up to compete even after the SNES released. Had the SNES released a year or 2 earlier, i feel like most people wouldn't have given Sega a 2nd look since Sega wouldn't have any major advantages over Nintendo to choose them over a well known brand like Nintendo.

Nintendo also took too long too release the N64, it was supposed to release late 1995 but again due to software delays as a result of its hardware being overly complex is got pushed to 1996. This gave the PS1 a 1-2 year headstart as being one of the only good 3D consoles that, by the time the N64 released the PS1 already had hundreds of games for the same price which made PS very enticing. Nintendo especially failed with this hardware transition not just cause they launched later than anticipated, but focused too much on specs rather than ease to develop for which is the reason why the N64 had so many huge software droughts from both 3rd parties and even Nintendo, and imo this decision to use complex hardware would hurt Nintendo going into the next hardware transition.

The GameCube also took longer than anticipated to release, Nintendo was determined to get the system released by 2000 to not give the PS2 any headstart but failed again since the software wasn't ready. Since the software teams were taking so long to produce N64 games on the complex hardware, it took longer for them to transition to creating games for the GC and based on what it seems it sounded like Nintendo's software teams didn't really want to transition from the N64 since they were just getting the hang of N64 hardware towards the end of the generation, they just felt sort of forced to transition. Even with the 1 year delay of the GameCube, after launch the GameCube had such a quiet and poor first year for the console when it came to games to the point where they rushed games like Mario Sunshine just to get games released in a timely manner. Nintendo definitely could've handled the transition better.

The transition from the Wii to Wii U could've been handled better, Nintendo was unprepared with HD development which took them much longer than anticipated to get a good release schedule for its first year.

Imo the best hardware transition has to be from Wii U to Switch, even though the Wii U's life was cut short, it still feels like Nintendo gave it a proper ending giving it big releases like Mario Maker, Splatoon, and Zelda BOTW to end its life and Nintendo was completely prepared to launch the Switch with one of the greatest first years of a console ever when it comes to games, a new 1st party game was release at a rate of nearly 1 per month, its one of the big reasons why the Switch succeeded so early.

I'd say 3DS to Switch was the best one for the reason that a lot of people didn't even realize that a hardware transition was on the way and instead believed that Switch was not a successor to the 3DS, but only to the Wii U. On the same note, Wii U to Switch was among the worst ones, first and foremost owed to the fact that the Wii U was on its death bed in its fourth year already.

As a general rule, any hardware transition of any console manufacturer runs a lot more smoothly when the current console has good third party support and is followed up by a new console that has either a strong first party lineup in its first year, a good third party lineup or, ideally, a combination of both. Whenever the games are there, hardware specifications matter little.

Rating each hardware transition of Nintendo with a simple Thump Up, Thump Sideways, Thump Down system, we get the following:

NES to SNES: Thumb Up. First and third party support for both consoles was great during the transition. The NES got goodies late, the SNES early.

SNES to N64: Thump Sideways. While the SNES coasted off into the sunset in decent fashion, the N64's early period was plagued was sparse releases and many delays.

N64 to GC: Thump Down. The N64's final year was bad, the GC's heavy hitters arrived a little too late and still felt rushed regardless.

GC to Wii: Thump Sideways. The Wii started strong, but clearly at the cost of the GC.

Wii to Wii U: Thump Down. Nintendo shorthanded the Wii and it ultimately still amounted to nothing much for the Wii U.

Wii U to Switch: Thump Sideways. While the Wii U was left to die for Switch to prosper (and rightly so), this isn't the ideal way to go from one generation to the next one.

GB to GBA: Thump Up. The GB's long run ended on a high note with two Zelda games, the GBA was flooded with games right away.

GBA to DS: Thump Up. During the year of transition, the GBA's lineup was notably better than the DS's, but the DS had a slot for GBA cartridges, so overall it was still positive despite the DS being rushed a bit.

DS to 3DS: Thump Sideways. Can't complain on the DS side, but the 3DS was a misfire early on.

3DS to Switch: Thump Up. Touched on in the first sentence of this post already. The 3DS had an extended period of continued first and third party while Switch's first year first party lineup was so strong that it makes one forget or forgive how damn bad third party support for it was.

In summary, Nintendo's track record covers the bases pretty evenly, making it easy to predict that the transition from Switch to its successor will be good at least on the Switch side. Worst case is something akin to the transition from DS to 3DS.

The transitions I rated with Thump Sideways did go two ways, the more positive outcome being the scenario of bad current console leading into good new console rather than the other way around.

Has to be WII U/3DS to Switch, turned Nintendo back into the powerhouse it was with a single device instead of two.

Nintendo's problems in part are due to software development starting too late I think in a lot of cases.

If I was running Nintendo I would have insisted (not asked, insisted) that the Mario Kart, Animal Crossing, and Smash Bros. teams must be working on hard development (not just planning, but actual level/world building, etc.) successor franchise installments of those IP for the next generation no later than January 2022 already, so the next installments of those games should already be well over a year into development already. I don't know what the Mario 3D/2D teams are doing but they should also be not only well into development of their next game but well past 50-60% complete.

You shouldn't even really need finished hardware for that as you can approximate what the next system's hardware would be like and use an older Nvidia graphics card to at least begin development on.

Everybody got a break most likely over the COVID break, and the last installments in these IP are now years in the past, it's time to get down to the nitty gritty of making the next franchise installments.

Gotta have content and the next transition will be harder for Nintendo because they're not going to have a Wii U library that was largely unplayed by a lot of people to pick apart and repurpose for easy hit titles in year 1 (ie: there's not gonna be a BOTW or completed Mario Kart 8 just laying around this time). 

Nintendo finishing Mario 64 in May-ish 1996 and still not being able to have a Mario 64 sequel ready for even November 2001 (GameCube launch in the US) still sticks out to me as basically like personal sabotage, lol.

Like what the fuck Mario team, what were you doing. 

This.