The issue with this idea is that you have to make something the baseline, otherwise you cannot guarantee that anyone with any specific model can play any specific game. There were already issues with the New 3DS and the potential split in the market, which is why only a handful of physical games were actually exclusive to the New 3DS (Xenoblade, The Binding of Isaac, Minecraft, etc). Even when the N64 came out with it's expansion pack the vast majority of games did not require the pack (only DK64, Majoras Mask, and Perfect Dark).
The second issue actually stems from the first issue. Since games would be made with a specific baseline to prevent a split in the market any hardware improvements that are provided wouldn't be as tangible. Of course increased RAM, CPU, GPU would help resolution, framerate, texture filtering...but games themselves would be limited to what the baseline could produce. Then you have the potential issue I'll call "The Outer Worlds" issue. That is, there might be a game like The Outer Worlds that is just complete dog sh*t on the baseline hardware. Maybe you think that is okay, but the people that are purchasing a Switch aren't quite as tech savvy as someone that might be building a PC. Seeing the quality of a game, calling to find out whats wrong with it only to find out that on top of the $300 console you just purchased it's recommended that you purchase a $50-100 upgrade or two for the game to actually run well. That's bound to give the average consumer a sour taste.