You're forgetting that that 20% of the gaming population (using your imagined figures) buys a disporportionately large percentage of the games.
All in all, I'd guess that 20% cores spending is roughly equal to the spending of the other 80%.
There's definite reason to cater to that market, plus it's easy. They'll buy the same damn game over and over and over and over again. Just up the graphics, tweek the gameplay, slap an incrementally higher number on it and BAM! you got sales! Or ape what someone else is doing, put a novel twist to it and bam! sales (just not as great).
The mass market is fickle and trendy. Few publishers understand their tastes. Most that do aren't making console games. They make phone, iphone, flash, social games where the mass market is (pretty much) the only market and they have to understand them. Originality and addictive gameplay is king here. No complicated controls, no big story, graphics not that important, but quick, easy, addictive fun.