Conker's Bad Fur Day - Escaping Teddy Island.
F-Zero GX - Racing against the Phantom Road A.I. (Only level that's basically impossible to 1st place on Master difficulty.)
Star Wars: The Old Republic - Darth Malak battle (and, to a lesser extend, Darth Bandon battle).
The Elder Scrolls V: Skyrim - Becoming a vampire.
The Legend of Zelda: Ocarina of Time - Finding out that you can see whether you have caught every Golden Skulltula in any given area after catching all of them. (Okay, this one is on me.)
Far Cry - Mutation encounters.
And yet, I still love all of them. Can't think of a case where an annoyance made me quit playing a game entirely. (As far as real games goes. Fuck Angry Birds.)