Yes they did. But I mention handhelds specifically because Nintendo's handhelds were getting closer and closer to the design and production values of their console games over the years, especially once they pulled out of the arms race on the console front, which culminated into the hybrid concept of the Switch.
That has, quite literally, nothing to do with Nintendo specifically. That's just the general trend of mobile devices getting closer to dedicated hardware in terms of power.
I think that the Switch becoming a hybrid device was through years of evolution particularly in where Nintendo's biggest markets were, as well as how unsustainable having two console to support was. I think that compared to those two factors, the similarities in production value was not as much of a factor, especially because that would be an irrelevant point had Nintendo made the Wii 3 with power similar to a PS4. I really don't think something like a 3DS compared to a Wii U is that dissimilar to a PSV to PS3, if anything the 3DS would have a bigger deficit.
That being said, I will say that if Nintendo had already invested in power to the point that the Wii U was comparable to an Xbox One or PS4, and it was at least somewhat successful (like 25m+) then... yeah ... maybe they wouldn't have gone down the hybrid route. That's one of the reasons why I don't think the PS5 will be a hybrid, at least not until many years down the line. And in a general sense, diminishing returns is a big reason why I think the Switch is a success. But again, that's not really what it seems you're implying specifically.