I think your assessment is more or less correct. A lot of companies wouldn't have bothered with ports/exclusives with significantly less RAM. Hell, it affected even the PS3 though the decrease in available RAM compared to the X360 was just a few dozen MB.
I doubt sales would have been more than a few million lower than it is (unless our alternate history lower RAM Switch also prevented the pandemic through the butterfly effect...) though it would certainly be a somewhat diminished console in terms of games library, visuals, etc.