Of those systems that missed out on winning their generation but still secured second place sales wise, which do you think was the best, and why?

Banjo, conker, Mario 64, jet force, 007, perfect dark, paper mario, etc. Blast corps, rogue squadron, etc. Great system, ushered a new era.

N64 is the 3rd best console of it's generation sorry, Saturn Japanese library is fucking amazing and so broad. N64 has some classics but less than 50 games owning tbh out of 200. All N64 had is 200 something games. Saturn have over 1000. A lot of great arcade games. Shmups. FIghting games. Platformers. Shooters. RPGs (N64 really lacked in RPGs) and the list goes on.

Overall tho best runner up. It goes to SEGA Genesis.  It's the first console to take away Nintendo's dominance and gave us the best console war. The first mascot to challenge Mario.  Each console specialized in something and was uniquely different. Even when they got some of the same franchises they were entirely different. Contra. Castlevania.  Genesis did not have as many great RPGs as SNES. But Genesis had some that held their own as well as any SNES RPG like PSIV. What SNES lacked in shmups Genesis more than made up for. While PC Engine is the best shooter console that gen. Genesis was damn good. There is a reason Genesis vs SNES is the most talked about console war over 30 years later. Sales were close and the console war was the most heated. Kids did not have multiple consoles back then. To this day an entire generation of people you were either a Genesis kid or a SNES kid. 

Genesis had the better beat em up series Streets of Rage over Final Fight. You had MUSHA. Ranger X. Phantasy Star. Rocket Knight and more that were just as good as what SNES offered in it's best fields. (SNES is my 2nd fave console ever behind Dreamcast btw) I'd argue SMW is the best Mario game of all time and better than any Sonic game but Sonic 2 is one of the best of it's era. Genesis remains currently the only non Nintendo console offered on Nintendo Switch Online. A testament to it's greatness.

Runner up was referring to sales, not quality, as that's subjective.

You didn't say sales and if sales all that count then PS3 wins as it sold more than everything else on that list. End of topic. Thanks for playing.

You're misunderstanding.

The question is, which of the consoles that came second in their generation (sales) do you consider the best? (Quality)

The N64 thing was a side comment but the real answer was saying Genesis of it''s generation. Separate from the N64 thing why it's in a different paragraph.

PS3. A multimedia giant and had tons of games, especially from 2009 onward. Had I gotten one before PS4, I would've played a lot. As is, it's a nice little machine to have for games I can't play on other consoles with a nice vibe.
I don't know if it's fair to call Xbox One better than Mega Drive/Genesis, N64, PSP and so on but it is an underrated console.

Gotcha; I have amended the OP to specify runner up in sales, for the sake of clarity.

It's a toss-up between the Genesis and the N64. While the Genesis had a lot of solid titles, probably more in terms of total volume than the N64, the N64 had more games that I'd personally consider among my all-time favorites. I'd put Super Mario 64, Mario Kart 64, Star Fox 64, and Perfect Dark in my all-time Top 20, while Sonic 2 is the only Genesis game on that list. Basically it comes down to the Genesis having more "good" games and the N64 having more "great" games. Both are systems I'd put in my Top 5 consoles ever.