Vanilla i think it was Dragonpriests for me. Of course, as others have already mentioned, after reaching level 40 or so it rapidly turned into a snoozefeast.

Recently started a new playthrough, this time modded to hell and back, and had to crank the difficulty and mod adjustments down abit since almost anything could kill me now.

I found that the whole sneak mechanic in Skyrim was too powerful, once you maxed out sneak and finished up with the dark brotherhood missions it was way too easy to single hit kill pretty much anything. The only thing it didn't help you with was the dragons, but they were pretty easy when they are on the ground. 

Those briarheart dual wielders were pretty difficult at times, so I'd probably give them my vote. 

Would you mind detailing these mods and adjustments? I'd love to start a new game and actually be challenged. I've concentrated mainly on visual mods, memory usage fixes and UI mods, hard to find good scaling and content mods in the sea of poor ones, sometimes.

With a high enough sneak and a good dagger and the Assassin Perk (I think that's the name) where you hit for 15x damage on sneak attacks; one-shotting dragons becomes routine, you simply creep up to them and strike. Even if you don't manage to outright kill it; you will at the very least very likely ground it for having removed the majority of its HP.

Yeah, I agree about the Sneak mechanic and bows in particular, really unbalanced mechanic. I love stealth gameplay of all kinds but not when it removes all sense of danger. The perks are what really topples the balance in Skyrim; the whole skill system allows for some really unfair advantages over enemies and NPC's.

I was looking forward to fighting a mammoth I once spotted, but it fell of a tiny ledge and somehow killed itself. =/

There are mods to get a real difficulty for this, but it seems a lot of casuals love W-RPGs now.  My expericience could be anecdotal tough.

The frame rate.

Relative to when you encounter them, I'd say those frost trolls on High Hrothgar. Ugh.

Not at home atm so here is just the ones i can recall right now that i currently am using/testing:

Deadly Dragons/Deadly Monsters: http://www.nexusmods.com/skyrim/mods/3829/?

Enemy Ai overhaul: http://www.nexusmods.com/skyrim/mods/40491/? not sure how good it will hold up further on as one advances up the levels though

I also use mods that while not cranking up difficulty on their own add mnore creatures and npc patrols, mainly the mods Immersive Patrols and Immersive Creatures.

while not using it myself i hear the skyrim Redone mod also can make combat more difficult since it rebalances weapons and armors

The children, couldn't even leave a scratch.