I think I would say if you look at games in isolation, the Sony screw up was the biggest, just because it was uncalled for and the results were cataclysmic. They didn't need the blue ray. Nintendo tried to have CDRom, they just messed up.
However, if you take into account that the the PS3 helped kill HD-DVD, in the big picture Sony might be OK with halving their install base if it accomplished the greater goal. In that sense, Nintendos screw up was bigger because it didn't accomplish anything at all, and cost them billions.