I think there is a combination of factors.
1. Pokemon has been on a steady decline for over a decade in Japan. Here are sales from GameDataLibrary.
2. The National Pokedex backlash is going to affect the "hardcore" crowd who pre-orders games months in advance more than it will with the average or casual fan.
3. COMG is rapidly becoming irrelevant when it comes predicting the performance of games.
I also think the size of the userbase has to be somewhat of a factor, even if there are a few exceptions (i.e., Splatoon 2 and Smash Bros. Ultimate). By the time X/Y released in Japan, the userbase in Japan was about 5 million more compared to the size of the current Switch userbase (though I guess it depended on how many of the 3DS userbase at the time had the XL/2DS versions). That's still a lot of 3DS owners to attract to when X/Y was being hyped.
But I think regardless, Sw/Sh will be fine. If there is no upcoming sequel or third version of the games next year, I can see the games have some legs until the next huge Pokemon titles. As we have seen, all of the games from Gen 6 and 7 have sold respectively this past fiscal quarter. So its not like once the new generation is around, the previous Pokemon games just get dropped.