I'm going to go with the Joker, he's just a scarier individual.
Bane is someone easier for Batman to "get". He's a physical force, something Batman understands. Bane just happened to get lucky that Batman had been out of the game for 8 years and was probably out of shape, because he got his ass whupped pretty soundly the second time they fought. And he's the standard League of Shadows crazy, Batman/Bruce has seen all that before and understands their MO.
The Joker ... all bets are off. Batman doesn't understand him because he's quite frankly completely insane. The Joker forces Batman to compromise his morals to beat him. You're talking about a character who quite possibly self-mutilated his own face. Even to characters like Rhas Al Ghul and Bane, they would consider the Joker to be nuts. The Joker is a character that wouldn't give a second thought to murdering a bus load of kids ... for Rhas or Bane or Talia, even if their idea of justice is horribly skewed, they still would think twice about that.
Even in the comic books, the Joker beats Robin to a bloody pulp with a crowbar and then kills him by blowing him up (Robin 2). He kidnaps Barbara Gordon after shooting her in the stomach, strips her naked and takes photographs to torture Jim Gordon with in the hopes of driving him insane.
