Damn, yeah. I'm changing my vote to RTS, the skill ceiling there is indeed beyond anything else by far. You can practice your way into those other genres ranks but you can't practice your way into being more intelligent plus like you mentioned there is skill involved to cause you gotta be quick. You ever try playing them with a graphics tablet, ya know a wacom with a stylus?

Man, having come from the NES to the SNES and dealing with both Castlevania IV and Turtles in time, which were so disappointing in the challenge department, was kinda harsh. I remember beating both in the first try, as I did Batman Returns, and was like "really, is that it"? Both were real walks in the park. Thank goodness for Contra III on hard, or Axelay, F-Zero, Super G&G and even Mario Kart on the hardest modes for keeping my enjoyment at the time.

I remember when I first thought I was good enough to play competitive Warcraft III and in some rooms I would just be SLAUGHTERED. It was my first serious commitment to RTS and have not played any other since, because it would take too long to get to that level. I even remember going into a room and a guy, out of the blue, saying: “give me control of your units if you want to win”. I was like WTF is this? He then said “do it, I’m really good!”. I thought to myself “ok, let’s see, as most this will cause is a loss” and he then proceeded to control mine and another guy’s units and absolutely DEMOLISHED the rest of the competition. I could not believe how he was doing those many things at the same time, and excelling at ALL OF THEM. Incredible. Then soon after that time (around 2005-6) I started my Masters and later my PhD and from then the only series I would put hours in the hundreds would be Monster Hunter. Even if I consider myself to be pretty good at MH, I guess these complex RTS games take the trophy. And I used to finish Ninja Gaiden, Contras (I, II and III) and Super Ghouls and Ghosts with relative ease in my best days back in the 90s.

For me personally, its bullet hell games, because I find them hard wether im trying to be good at them or even when playing them casually.

They're designed so you have to learn by trial and error which makes them so difficult to master but the more you play the more you feel you've seen this pattern before, ah I'll maneuver this way and that to avoid it. There's only so many patterns they can code into the genre. 

Trial and error until you learn patterns is a mechanic we find in many games, the thing is theres way too many things happening simultaneously, and thats what always gets me. By comparison, learning the patters and getting gud in soulsborne games is way easier, at least to me it is. 

For me it's RTS.

On the other hand, the NES Turtles game that's often bashed for it's difficulty... well, I don't find that one particularly difficult at all until the Technodrome, and even then it's manageable if you don't rush (which I think is the main reason many find it so difficult, they hurry too much and get overwhelmed).

Like I said before. It took 12 years for someone to actually beat one of the DoDonPachi games. TWELVE YEARS! Because the genre can be so insanely difficult esp certain label games. It's a genre everyone can play but very few can master.

This is not a one-off occurrence either. One took 7 years another took 10. Everyone can play Ikaruga but less than 100 probably truly mastered it.  Border Down and Battle Garegga punish you for playing well so the games become almost impossible so people who mastered those games figured out when to strategically die to make the game ease up for the moment. Tho I mostly talked about bullet hell not all the most difficult games in the genre are bullet hell. Some are 16-bit era shmups. Gaiares is pretty tough.  Not enough people esp here experienced this genre enough but it's one of the most  fun yet infurating but rewarding genres.

Adventure/VN/Deduction games are the best at racking up racking up your brain in the most literal sense.

I mean, apparently, gamers do hate to read 🤭

But joking aside,
To me rhythm games are quite the most challenging kind of games based on pire skill and twitch reaction I could think of

Hehe. Good point though, I forgot about puzzle games. Tunic and the witness had people sketching stuff on physical paper like they were goingbthrough a manic episode and were seeing reality for what it really is. I don't know of you'd call it a skill ceiling but there's a summit to that kind of gameplay many people won't be able to climb to. Alas, it's only the singular games you'd be getting good at, I doubt skill from the witness is going to help you at all in tunic and vice versa aside from perhaps making you a little sharper minded.