Not all games should be designed for everyone. Instead, some games should be directed towards beginners while others should be more challenging. For example Nintendo: if you want an easy game, play Kirby. If you want a moderate game with great learning curve, play Mario. If you want a die hard platformer, play Donkey Kong Country Tropical Freeze. I believe that some players enjoy challenging games more than just watching everything roll over the screen without much need to interfere. That's why the Souls game are so popular.
As for older games and their counterparts these days, I think the several genres should be regarded differently. RPGs were very grind-demanding back then, which they usually are not anymore. I think that's good. Platformers have been very hard but it got differentiated as time went by. There are plenty of easy platformers and still some hard ones. Something for everyone. Shooters have also been extremely difficult, sidescrollers as well as run'n'gunners like Contra as well as early fps like Doom. Nowaydays it seems to me that shooters have no challenge at all anymore, instead they try to sell a Hollywood-like experience. But still, some shooters are very hard and challenging. So, everybody should find something for his needs and urges, I guess.
A side effect of super hard games back in the old days existed, too. It wasn't simply a form of entertainment, it was a life lesson. Games back then told you, that if you want to achieve something, you must practice! Oh, you're no good at this? Then sit your a$$ down here and get good! The feeling of satisfaction and relief that you only get when you managed to beat an ultra heavy and demanding boss/area is priceless. You cannot get that in easy games or easy difficulty settings.