"Casual games = Mainstream games that I don't like." is a common mindset imo.
Games themselves aren't really casual or hardcore but they are targeted against people who game more or less frequently. Some games (2D/3D Mario, Call of Duty etc.) targets both groups.
Categorising accessible games as Casual is extremely flawed as that would put Megaman, Metroid and Ghost & Goblins in the Casual bin just for having easy to understand controls.
I really don't understand why gamers are so obsessed with labeling games as casual or hardcore. Unless you develop or publish games that's really not something you have to think about. Elitism is one answer I can think of, as it reminds me a lot of the snobby extreme Black Metal crowd who labels any band selling more than 666 copies of their albums as mainstream crap, no matter if their music is good or not.
It's popular among fanboys as it's an easy way to discredit games (They are not real gamez!!!! 1337!!!) that sells well. It's a casual game and therefore won't count as a Victory Point.
