Because it's still a success. And they wouldn't limit themselves. There's plenty of time for Mario Kart and other new IP to come from Yabuki and the team. But ARMS is a successful new property that can grow a little with another game before all that. 

Let's look at it from a business POV; Nintendo in the MK8 squad have a team that is capable of making AAA 10+ million sellers. To then handcuff that team to an IP that's proven itself lacking in longevity and real hardware/software selling potential simply makes no logical sense.

What would make you think they'd be handcuffing them? ARMS 2 will be happening because Yabuki wants to make it, not because Nintendo mandated him to make it. If ARMS 2 was to happen, Yabuki would ask upper management for approval, and Takahashi's response would be summed up as "Meh, do what you want man, just make sure it's profitable". If the Mario Kart team was Nintendo's most valuable team at EPD, wouldn't make sense to give them a lot of leeway in creative freedom? 

Creative freedom is one thing, flushing potentially hundreds of millions of dollars down the drain by sinking resources into a game without real sales potential is another.

Again, ARMS sold 2 million copies, and still has an active player base. That's enough for a sequel to build upon. It could be as successful or even more successful than the original. You seem to think Nintendo enforces some imaginary policy where every game EPD makes needs to be the next mega-hit title or else no more games in that series will be made from them. That's not true. Nintendo doesn't care what gets made as long as it's a good idea with sales and amusement potential. ARMS fit the bill, so Nintendo's will to allow a sequel if Yabuki wants it. 

ARMS fizzled out within a year in both sales and mindshare though, no competent management would greenlight a followup to game that demonstrated such a severe lack of saying power.

ARMS only needed to last long enough for Nintendo to get all the updates out. It's a new IP in an already relatively niche genre, and the team only planned for 4 post-launch characters, with Spring-tron being added to the roster purely because the game was more popular than anticipated early on.  It sold well all through 2017, and even today, still retains a solid player-base. So a good year of support is enough for Nintendo in this case, and the game provides a solid foundation to build upon for the future. 

It doesn't offer a solid foundation though. Its sales curve and how quickly it was forgotten are both indicative of a game that has no long term potential.

You're acting like the game should be lasting the entire lifetime of the Switch in order to be a success. That's not the case, ARMS lasted long enough for Nintendo to consider it worth the effort. It sold 2 million copies in a year, and in a year enjoyed a healthy player base. That's enough for a sequel to be made, and since Yabuki wants to make a sequel, then there's a good chance that we're going to see one sooner or later. 

We don't know if Nintendo thinks it was worth the effort, they would never admit it if it fell short of their expectations, they pretended Wii U was fine for years.

In the end, it all comes back to one thing; prioritizing a 2 million seller over a potential 10 million makes zero business sense, and zero common sense. That's all there is to it. It could only happen as a vanity project okayed by incompetent management.