Disagree somewhat on this the reason Nintendo's IPs have had long staying power is because of changing up the foundations of each installment and taking the games into multiple directions one of the series that highlights this the most is Zelda as each installment utilizes a different idea with some even changing the entire approach and foundations of the series (ALTTP, BOTW...) the result is a series that has not only allowed for room to experiment but has been able to develop both lore and characters in a flexible manner to the point each game is another unique incarnation of familiar characters. Mario is the other best example of this with the mentioned Luigi's Mansion, part of the charm and appeal is the use of Luigi and the Mario universe as just like with Zelda it's a fresh incarnation of a familiar cast that helps broaden appeal even more fore them.
The use of an existing IP helped shape the lore and settings of such a project in itself and as I said a bit earlier about developing character LM gave Luigi a more defined identity rather than being who player 2 plays as, his personality and popularity effectively stem from taking an existing IP and expanding it this is more a case by case thing than outright that should be a new IP scenario.
Sure, and that's something Nintendo is great at. But one of their most stringent design philosophies is "Form follows Function" Most everything in a Nintendo game exists to fill a game-play or interactive need. From characters, to aesthetics, to music, visuals, etc. Sure, you could slap Mario and Friends into ARMS for example, but then you'd have to come up with a silly, contrived explanation as to why they suddenly have Stretchy arms and masks. It wouldn't be marketable or endearing, it'd just be forced and lazy. In situations like that, you need characters and aesthetics that match the game design you want to create, and that's why we ended up with a New IP vs. another Mario spin-off. When Nintendo does apply a pre-existing IP to a new project, it's because the game design actually reminded them of an IP that could work really well with what they're making. Nintendo doesn't typically start with IPs, they start with Prototypes and attach IPs to the ones that get greenlit. Be it fresh IPs, or pre-existing ones.
Nintendo has traditionally gone out of its way to try and avoid redundancy in their stable of IP. Feeling that Each IP and game should be radically unique from each-other and others on the market, even within the same genre. Mario looks nothing like Zelda, which looks nothing like Splatoon, which looks nothing like Animal Crossing, which looks nothing like Fire Emblem, which looks nothing like Brain Age, which looks nothing like Kirby. The thinking is that "Unique games = Profit" so why waste time on something that plays mostly like a Zelda game in everything but name, when you can instead just make it a Zelda game and save time and money, while making something with much more sales potential.