No, the worst was when the N64 had to compete with the Psone
Nah. I had both. You have to remember that while yes, N64 got it's asses handed to it sales wise, that was the console that gave birth to SM64, OOT, MM, 007... it had some games on it that completely changed gaming and revolutionzied the industry.
I'd say the worst time to be a console owner was the Gamecube era. Compared to the N64 titles, most of Nintendo games like Sunshine and Double Dash were no where near as amazing. There were a few gems like Metroid Prime and SSBM, but for the most part, it was rough.