Sub 10k and sub 2.5k for the two SKUs in week 4 isn't anything impressive.
It's lower than Nintendo Land and 1,2 Switch and Nintendogs + Cats at the same point, and I wouldn't get too attached to this game in the top 5, it's there because it was a slow week once other games start releasing, odds are you're never going to see this Labo Variety Pack in the top 5 ever again (Robot set never ever made the top 5 period).
And again, your looking at Labo as a traditional singular game. Over 10k for both kits combined, and still charting well post GW is still pretty good. And We still haven't seen its performance in the US yet, nor have we seen the sales from Nintendo's perspective. So it's hard to really call it a failure when there's nothing to really base that on.