The DS Lite was overall a fantastic revision but its one significant flaw is its durability. That weak hinge made it so it was not difficult to break so as a kid I went through like three of them while I still have my GBA SP that kept working despite getting pretty banged up.

Best
GBA SP: The gold standard.
DS Lite: Cheaper and brighter, which is even better than the slimmer nature.
PS3 Slim: Yes it lost PS2 support, but later SKUs of the Phat PS3 already removed that. PS3 Slim was a fantastic price and turnaround for a console that was turning around to an extent already but could've sunk.
Xbox 360 S: Built-In Wi-Fi and Kinect support, a better slimmer build.
Xbox One S: Slimmer with a 4K Blu-ray drive added.
PS4 Slim: You lose really nothing from the PS4. The face plates I guess but few used those.
Worst
Wii Mini: It took two things revolutionary for a Nintendo console at the time (backwards compatibility and internet functions) and removed them, among other things.
PSP Go: Paltry internal storage and too ahead of its time for digital distribution only. UMDs sucked but removing them from the equation for a PSP was not a good idea.

Some I'd throw in:

Win:

PSOne

Tiny enough to take with you and cute as hell

Gameboy Colour

Massive step up over the base model

Xbox One X

Solved the original model's power deficit by one-upping the PS4 Pro, and looks cool

Fail:

2DS

Looks like some bootleg shit from temu

PS5 Pro

$750 is an insane price and the power bump is unimpressive. Also still looks overdesigned and tacky

PSP Go

Losing physical media was a massive fail at a time when digital hadn't fully developed.

It was meant for 4 year olds. Making it as unbreakable as possible.

I'm sure they could have come up with a design that was cheap and sturdy that didn't look that ugly though.

This will be controversial. But I like the original GBA more than the SP.

It's the form factor. To me it is more comfortable to hold. The SP and other clamshelves hurt my hands with longer playing sessions.

Same. I'm not against clamshells but I don't like the GBA SP design. Would have preferred the standard GBA got the frontlit, and eventually backlit, screens.

A shame because its slide up design really should of been the future for Playstation handhelds

A bunch of stuff I would mention has been mentioned.

So maybe not the best but it was an improvement. Neo Geo CDZ. The Neo Geo CD was notorious for extremely long load times. Sometimes up to 2 minutes! You saw a monkey juggle so there was that. CDZ put a 2X drive in the new model and games loaded much faster. 

I will throw in Wonder Swan Color & Neo Geo Pocket Color. Libraries much smaller tham Game Boy but like GB to GBC it helped games a lot. For a while Wonder Swan Color was the only portal way to play some of the 16 bit FF games.

One of the worst is the last PSP model. Like Wii Mini. They stripped it of anything useful for the time. It was barebones.

I have a soft spot for the SNES Junior as it's just so little and cute.

The Megadrive Model II also looked so sleek and badass