The problem with all those smartphone and casual gamers is that they are not dedicated to games in any way. They might enjoy Candy Crush for some weeks, but after that, more often than not they won't look actively for a new game.
The core gamer on the other hand has gaming as his main hobby. He constantly buys new games at a relatively high price point and you can be pretty sure that he will be gaming even in 10 years from now. So, your casual folks are always good for a quick cash-in, just because there are so many of them. 1 $ from every iPhone user on the world? You're goddamn rich! But if you piss off your core gamers too much, most of them will leave you. It might just happen that sometime, you find yourself without any supporters left. That's when you're taking the road to benkruptcy. Depending on how much money you have in the bank, that might take a few months or even more than a decade. Doesn't really matter, it's a dangerous road to take. You should always focus on your main audience, your bread and butter. Of course you can take a look for new markets, but never lose focus of your core business.
A pretty good example imo is Nintendo. They took the world by storm with the Wii and the DS only to fall even deeper than with Gamecube with the Wii U. The casuals left and a lot of old Ninty Fans were pissed off and simply didn't want another Nintendo system.
There is no black and white. That also applies to the gaming industry.