In life, it is a matter of balance. I have a gift involving games, so I am involved with them. I know a lot about them, and seek ways to do things with them. For me, I believe games can be used for greater purposes, just like any art or craft and things would benefit from doing so.
It is easy to beat oneself up over how many years they spent their lives on games, but should I argue the same for my getting a Masters degree? That had saddled me with a lot of debt, and doesn't show a sign of paying off now (out of work several years in an industry that doesn't want to hire people out of work for over 6 months). For me, I am finding sanity in working on a non-profit that promotes chess and other abstract strategy games. It gives me a place to express myself and do things. It also forces me to deal with myself and become a better person.
Now, if you happen to bury yourself as just a consumer of games, and have it as totally self-gratifying, then you will end up possibly in a very shallow place, and go through a mid-life crisis.