I almost always start on Normal. I feel that, for the most part, that's the mode that's been balanced and tested the most. Also, honestly, the harder modes are sometimes just cheap and annoying.
If I really like the game, though, I'll often play it again on Hard, especially if it's a game with an unlocked + mode. Borderlands 2 and Torchlight 2 are two recent games where I jumped right into the next difficulty. That's something that I'm glad has gotten popular, as I love expanding on what I've already done.
I do like the option to drop difficulty down for really annoying bosses, though. Seriously, some aren't very fun at all and I just really want to get past them. I did that yesterday with Deus Ex: HR because I just was NOT feeling that boss fight.
