If I had to rank from biggest failure to smallest failure, I would go Saturn, Xbox One, N64 & PS3.
Saturn is the biggest failure of the four because it basically led to Sega leaving the console race. None of the other three have led to that. Xbox One was a pretty big blow to Xbox though and they still haven't recovered. N64 is next because it was the first time Nintendo wasn't in first place in the console race and they lost big exclusives for a while. They were saved by the fact that N64 still had iconic releases. Also, they had their handheld market to fall back on and Pokemania. They also turned to a blue ocean strategy that was a mostly successful. I consider the PS3 to be the smallest failure of the three because even though Sony lost billions on it, it still sold the best of the 4 and Sony didn't have to completely change their strategies in order to course-correct.