I feel that way every time I go past a store that has Switch on display, mainly because of these observations, which are of course related only to Japan and I can't speak for how things are going in other countries since I'm not there, just first hand experience with Japan in particular.
1) The initial buzz around launch faded exceptionally fast, the low shipments nation wide caused a dearth of stock on auction site being flipped for a profit, but even with that, mere weeks away from the consoles launch and I don't see anybody standing by the displays anymore, hell last night I went to see what was new and a bunch of kids were in the store, some playing a game on the display unit vita, and some playing on the new mario sport game for 3DS, nobody, at all, was paying attention to the switch stand or the switch section of the accessories isle and I stayed for a good 2 hours talking to a friend who works at the mobile phone store right along side the games store (all open plan, so the shelves just seem to blend together).
2) The scalpers who snapped up the initial stock have had a hard time shifting the stock for any worthwhile profit, normally by now all but the most insanely marked up consoles would be gone, but as it stands auction sites are still flooded with the consoles, many asking for little more than $30 over standard store price, which says a whole lot about demand for the console currently.
3) Both in-store and on TV, advertising for the Switch has been wishy-washy, for example in an hours worth of TV, you'll maybe see one Switch commercial which doesn't seem that great of a commercial to begin with, conversely you'll see several adverts for 3DS games (such as the new mario sports game) and many, many more mobile games such as granblue fantasy, puzzle dragons and so on. Basically the consoles presence seems overshadowed by both mobile and it's much more popular brother, the 3DS.
4) The general response I get from people when talking about the switch is that because it's advertised as a handheld, the price is too high, if you compare it to the 3DS which averages around 18,000 yen and the vita which averages around 17,000, for the Japanese, there being another handheld that is 32,000 yen just doesn't seem like value for money - regardless of technical spec, the "its a handheld" portion of the consoles philosophy is detrimental to its value proposition for Japanese buyers.
Likewise, with 3DS games averaging 5200 yen, seeing the marquee title for Switch, Zelda, costing upwards of 7300 yen only compounds the situation brought on by not seeing the value in a 32,000 yen console.
5) Zelda just isn't a popular in Japan as it is in the west, I know that sounds crazy since it's a franchise born out of Japan, but the fanbase for Zelda has shifted it's core from Japan to US and in some ways, the EU, and I think it's largely a generational thing, gamers, even older gamers in Japan associate Mario with retro gaming, where as in the US, Zelda is seen as being one of the highlights to retro-gaming memories almost as much as Mario, with faithful and loyal followings. - Don't get me wrong, there's still plenty of Zelda fans in Japan, it's just nowhere near as common as it is in the US, as such, the only (let's be brutally honest here) launch title worth picking up for the switch right now is Zelda, and when the appeal for that isn't super strong, and you discount it from the launch lineup, there's little much else left behind to pull an audience.
6) Smartphones - everyone has them, be they the 75 year old salaryman on the train of the 10 year old walking to school, be it IOS or Android, everyone has them and a large majority of them play games like puzzle dragon and digital card based games, the steep rise in attachment to smartphones as both an internet portal and a gaming device has taken a large bite out of the gaming market, and is causing generations of gamers to grow up having only, or mainly, experienced gaming on such devices - in 10-15 years, this generation will look back to old smartphone games they played rather than standard console games, and this has, and will, continue to impact the success of consoles in the region.
Because of all of these reasons, and without any experience of the console outside of Japan, i'm inclined to cast a rough estimate that the Switch will struggle to break 50 million lifetime.