Which system do you think represents the biggest improvement in quality over its predecessor, not in power but as an overall experience, game library, etc, and what makes your pick so much better in your opinion?

Saturn to Dreamcast. Saturn as much as I love it thanks to the huge and amazing Japanese library. In the west it was lackluster library (some greats) and the US launched with a different controller than Japan. It sucked. System had a big clunky memory card slot. Also used for ram expansion.  Saturn itself was kinda big and ugly. Dreamcast was sleek. Futuristic looking. Cool VMU memory cards. A slick UI for the time. It just worked. Online being built in. Analog sticks out of the box. A demo disc. Web browser in the box. 

N64 to Gamecube. While personally not a huge fan of the Gamecube controller. It was a huge step up from the N64 pad. The neat trigger buttons. Gamecube looked neat. Oozed personality. Simple but sleek UI with the Famicom Disk system music played in slow motion. Not as cool as the PS2 memory card icons but Gamecubes were neat. Before it set in how limited the discs were. The fact the games were on discs was cool. How small they were was neat. GBA Player was great! 

Xbox to Xbox 360. Similar to Saturn to Dreamcast. Everything about it went sleek and futuristic experience. 360 took the media features of Xbox and expanded on them. Win Media Center. Connect MP3 player. The blades was the coolest UI console idea then and now. For many Achievment points was an interesting and novel concept then. User rankings. Similar to Dreamcast you got a free puzzle game right away. Sleek online features. 

Gamecube to Wii. Wii was so unique. The pointer just worked. The channel concept was novel and fun. I spent a lot of time in the news ones to see the cat. Mii Votes channel. You could have fun with the channels before you play a Gamecube or Wii game. Not to mention how sleek and cool Wii looked esp with the blue pulsing light. 

Wii U to Switch. Wii U was a half step to the Switch concept. Wii U was clunky in looks and in function. The UI ran slower than a 3 legged turtle crawling in a puddle of molasses. Switch cut the fat of that UI. Back to something bare bones simple but was fast, Clean. The system was the finalized idea Wii U tried to achieve. Switch was simple but elegant in function hardware and software. 

PSVR1 to PSVR2, from cable mess, bad controllers and tracking woes to being in the game 30 seconds after turning the PS5 on.

I think for me it was the original xbox, it seemed such a huge jump from the previous generation and some of the games still feel competitive in gameplay and graphics to today. It also had the duke controller which I loved with my big hands which is still perhaps my favourite controller of all time. On the portable side it has to be the Switch because it was the first portable console to allow big ambitious games on the move. I think recent generations have been disappointing in that they provide marginal graphic fidelity improvements but no radical departure from what came before.

Even though it's the same system I say XBox One to XBox One S. The One S got a slight performance boost no longer has an external power brick and the disc drive got upgraded from a Blu Ray drive to an Ultra HD drive. Plus, it's smaller than the X1 even though they put the power supply inside the console.

Ps1 to ps2

I have enough experience with PlayStation consoles only, so I'll say PS2 to PS3. PS3 enabled open world games in a way unlike before, and janky gameplay (e.g. because of poor cameras/controls) largely became a thing of the past. Online play was also enabled for the masses (at no extra cost, I might also add). Some might also appreciate the wireless controller and Blu-ray movies. I'd also mention motion controls, but I don't think they were utilized very well in most cases. Oh, and digital distribution is probably something many appreciate as well! There's downsides as well, like limited backwards compatibility, but I think the net improvement is still quite good.

Reiterating as per the OP, that this is not about the graphical leap, but rather being a better overall system.

PS2 to PS3. The PS3 could do just about anything, including running Linux. Sure, they scrapped that a while later, same as PS2 compatibility. But when I got my PS3, it was packed with features and functions. The step to PS4 honestly felt a little bit like a downgrade to me with less features and paid online gaming.

Wii U to Switch comes to mind. The Wii U was such a slow and sluggish-feeling system, long loading times, etc. The Switch felt way more modern in comparison.

Wii was also huge improvement compared to Gamecube. System felt so sleek with its new controller and design.

PSP compared to any handhelds before and also the original DS model. The PSP was an extremely modern, state-of-the-art multimedia system with this huge screen and incredibly slick design.