The amount of people not playing games vastly outnumber those who do. Of those who do, the number playing games on something other than a console is in the billions.
In the larger picture, console gaming is definitely a niche.
You could say that about everything almost. There are billions of people that don't watch soccer, football, baseball, basketball. There are billions that aren't *insert specific religion*. Are these all niches?