I voted for Half Life 2 as when I think of mixed of impact and quality it deserves game of the year. Looking at just impact and culture alone World of Warcraft clearly had the largest. There people who never played a MMORPG in there life that I have heard use the Leeroy Jenkins meme.
My Personal Favorite game of the year is Vampire: The Masquerade amazing game that I go back to and play again ever couple years. I decided Half life 2 deserves game of year a bit more but Vampire: The Masquerade is the game I personally enjoyed most. Half Life 2 would be my personal second favorite game from year. World of Warcraft I played for about a month but just was not something I could get into and never played it again after that.
Dragon Quest 8 would be my favorite console game from year but in 2004 I was basically only playing PC games. I did not even own a console from this generation. I ended up burrowing my brother PS2 to play Dragon quest 8 in 2006 I think and really enjoyed the game.