By far the Play Station controllers for me. I really dislike the layout, even if the quality of the last one is really good. It actually hurt after playing with them for a while. I have Horizon and love the game but cannot play for too long without my hands hurting at some point.

After that I would say the Sega Genesis/Megadrive one... The shape was so weird and the 6 button felt really weird to me.

Finally I would say the original fat Xbox controller, this thing was just so enormous... Still played a lot with it until the small one came out and even ordered one from Japan (for a reason it was available there first).

Yeah, I have an Intellivsion II, and the side buttons are so stiff that a lot of times it tricks me into thinking I didn't even press them.

The only controller I truly did not like was the duke. When I 1st tried it, I was still quite young and small, so my hands could not use this beast of a controller. The buttons felt odd as well. I ended up not being able to play whatever game was being demoed, and hated the controller as a result.

N64 and original XBOX.

Ignoring the N64 controller (which i can only assume was developed with something other than humans), i had some problems with the PS3 controller. By the time i got a PS3, my hands had grown too large for it to be fully comfortable. I still liked the controller overall because i prefer the analogue stick placement, but damn if i wasn't happy when i saw the DS4.

PlayStation 1,2 and 3 controllers. They're just so small. Analog sticks too close together and mushroom like.
Wii mote. Pretty similar reasons. Lack of buttons too. I got a pro controller quite quickly after I bought my Wii.
The duke. While I found it comfortable, it simply was just too big.
Atari paddle controller. Wtf. Seriously, wtf. It worked well for the time. I just hated it. For almost no reason. I'm allowed to blindly hate some things? Right?

A real gamer has no limits and overcomes all obstacles.

They all have their positives and negatives. but you get used to them quick.

It's one of my favorites. That d pad was horrendous though. I play fighters with the d pad. It cut me up and I ended up taking more damage than my opponents.

It was better in later versions but the damage to my body was done and had to retire. My body was never ready again.

Actually, everyone here has failed to mention possibly the worst of all time, the Atari 5200. The failure rate of those controllers were so high, that many of them broke after mere weeks.

Since I see it coming up alot here, as far as the N64 goes I never had a problem with the official N64 controllers (save for the torture on your palm from the Mario Party titles), but one year during the holidays I asked for an extra controller and ended up getting this disaster instead...

If you thought the standard N64 controller was awkward, try playing games with this abomination.  The buttons were clunky and would stick after awhile, the Dpad was even stiffer than the standard controller, and the analog stick wasn't nearly as responsive.  Plus the cord was cheap and within a year the wiring at the back of the controller where the cord connects became completely frayed and I ended up tossing the thing.