I'm sure Nintendo can do it, cause after all it's Nintendo, but that's the problem Nintendo needs sales to become inspired and motivated to give us another Super and they gave up with Zero Mission and outsourced Metroid ever since.

Samus Returns proves that Sakamoto still understands Metroid, it's not as good and complex as Super but it's a remake and a respectable entry in the series. There were some rumors about another 2D Metroid in development for 3DS which seems very unlikely at this point but im sure it's  going to be Metroid 5.

I wonder how they are going to do it with Prime 4 and Metroid 5 both releasing on Switch. We might actually get a Metroid (Dread) 5 before Prime 4 now.

I feel like they're finishing up something that's going to be both revealed and released this year and they're in the process of gradually transferring staff from their 2019 game to prime 4

My bets:
They have been working on star fox racing since DK. Took longer to develop because they had to move to switch, ad then ignore ideas of gamepad.
Now it is ready, but due to starlink and some racers on console (crash, sonic racing) in late 2018 to mid 2019 they are postponing the reveal and the launch date.
We will get a E3 reveal and a 2020 launch date.

Ermm...

Within 5 years they released 3 Metroid Primes (2002-2007), in the following 7 years they worked on an additional 4 new games (MK7, DK Returns, Tropical Freeze, Meroid trilogy).

So 5 years in and no new game to show, speak of or be credited in is absolutely not normal. All logic points towards them having a big game on the Horizon hence them never taking on Prime 4 to begin with and hiring like mad. It would be a huge failure on everyone's part for them to have spent the last 5 years on cancelled projects and not releasing a new game til 2021, that's kind of unprecedented for an internal Nintendo Developer.

Here's my theory on what has been going with Retro for the past 5 years. After Tropical Freeze was completed they began work on a new game for the Wii U (let's call this dead game), A couple of years into the project (Let's say late 2015 or early 2016) they decided to split into 2 teams as Miyamoto alluded to back in 2013.

https://nintendoeverything.com/miyamoto-can-see-retro-developing-multiple-titles-simultaneously-studio-is-a-very-high-priority-for-new-potential-metroid/

This second game becomes Star Fox Grand Prix and was never a Wii U game, but a Switch game from the start. Some time afterwards Retro moves development of dead game to the Switch like in spring of 2016. But some time after the decision was made to port dead game to the Switch, they start running into major development issues with dead game has a result it becomes a in-salvageable mess while Star Fox Grand Prix is shaping up to very promising and development is going well, so the higher ups at Retro and/or Nintendo cancels dead game to put more resources into Star Fox Grand Prix to make it better game.

Late 2018, as Star Fox Grand Prix becomes content complete and enters the QA stage for its reveal and release in 2019, Retro has internal discussions on what their next game so should be. They perhaps got word that Prime 4 was having a rough development, so Retro made a pitch to the higher ups at Nintendo to have at least some of their staff get involved by making a demo that impresses Nintendo way more then at any point with Bandai Namco has done with Prime 4 these last couple years. So Nintendo decides that it is best to let Bandai Namco from the Prime 4 and let Retro be the main development, Retro decides just to scrap and start Prime 4 over due to things such as having trouble working with the code and having a different vision for the game.

To develop for the sixth generation is not the same thing than developing for today's market.

1-2 year development cycles were the usual all the way to the early 2000's. Now it's easily double or triple that.

Think even back further, when FFVII took a single year to develop with 100 people. That was unprecedented in scope back then, and now it's your average DLC or season pass development group.

Also let's say Retro decided to make a new engine since Tropical Freeze runs on the same as Prime 1 and became incredibly outdated. That could by itself it explain why they have been quiet for so far long has some times it could end up doubling the length of a development cycle, as the engine and game are being made at the same time. Which is why you see so many games either have UE4 or a company wide engine like Frostbite since building a new one takes such a long time. 

This is true but the point is more the fact that their development cycles and processes haven't been particularly distinct from the rest of the industry.

Cancellations happen, often in quite but even by todays standard 5 years of development & nothing to show is not normal. Even the biggest most expensive games like RDR are 6 years full cycle, so if Rare somehow doesn't produce a AAA title in the next 12-18th months and their next title is infact Prime 4 which isn't gonna release for another 3 years, that would be a stain against their reputation and far cry from how they've been managed in the past. 

I think Retro split up into two teams, one made an unknown game and the other made Star Fox Racing, I don't think the spilt occured in 2013, right after the end of development of Tropical Freeze, but probably later on, like 2015 or 2016 when the size of the team was already larger, perhaps even due to issues with the game we know nothing about.
The Star Fox game development proceded smoothly while the other game had a troubled development. I think the Star Fox game is ready and will soon be shown and released this year, moreover due to the rumors about Retro pitching a demo for MP4 to Nintendo I'm going to guess the Star Fox team worked on that demo, possibly using some assets from Star Fox GP since it's also a Sci-Fi game. The other game was either cancelled or put on hold for the time being, as I expect both Retro teams to merge back to make the development of MP4 faster.

Am I the only person in existence that hopes this supposed starfox racing game is just a bad dream? Sounds terrible. I’d rather see a new Donkey Kong game than that. Actually it’d make sense cos I’d think DK would be a series Nintendo would want them to continue with. I also think that’s (partly) why they rereleased dktf on the switch.