Depends on genre.
With most action games I prefer casual, just playing the easiest.
RPGs definitely need to be easy and casual.
In puzzle and adventure games (point'n'click and such) being too easy makes it pointless.
Strategy games depends, if there's a campaign to fight through I'll do it easy, if it's just a general thing you play again and again then I go up to about medium to see what level I can still beat.
Driving games are ones that I most need difficulty with. Though in suitable amount, I do prefer to be able to finish games.