Working on a new IP. I guess it will come out next year. Hopefully it's something big too.

Maybe they're working on two games simultaneously, and in doing so ended up biting off more than they could chew, slowing down both games' development. Or they're simply working on a new IP that has been more problematic in bringing their vision to reality than they had anticipated. They have only ever been accustomed to working in the confines of existing IPs after all. 

Miyamoto said that Retro is big eunf that could work on two project simultaneously, of course they will have own game, most likely Retros game will be released in 2018. (offcourse that Nintendo didnt show all their 2018. games).

Those two games could be projects like Mario Kart 7 though where they're assisting another studio. Before E3 I would have agreed with you, but after three and a half years with nothing to show for themselves, I'm really starting to doubt they even have a big project on the go. They've never gone this long without announcing a game before.

To be fair, DK TP was finished by october 2013. Nintendo pushed it back to pad out their early 2014 lineup.

So technically that's less than 3 years between DKC Returns and DKC TP. They've been unaccounted for for nearly four years now, with not so much as a hint as to what they are working on.

And usually while deveopment for a game is wrapping up, a small core team is already working concepts and prototypes for the next game, so that develpment can start immediately. We also had all those reports of Retro hiring and splitting into two teams, so it's really wierd that we havent heard about anything they've been worknig on.

My guess is that they probably started developement for whatever they're working on on the WiiU and that it got pushed on the Switch at some point, so development would take a little longer. I really exected them to show something Retro this E3 though. Since they didn't I'm thinking their next game might be a Holyday 2018 title, wich would mark a five-year development span.

They announced back in early 2014 that their next project had begun development in late 2013 for the Wii U and that it was not another DK game or a Metroid game. After that, nothing was announced, though Retro had been active on Twitter prior to E3 events over the years leading to speculation that they might announce something, only to announce nothing.

If I had to take a guess about what's going on, I think that the project they were working on was most likely put on hold due to poor Wii U sales. The project was then most likely moved to the Nintendo Switch or may have been shelved altogether. The latter most likely didn't happen unless the game made liberal usage of the Wii U's Gamepad, something the Switch doesn't focus on so much.

development hell I guess

You're gonna need to provide a source on this, because I've definitely never heard that.

We definitely know that it is not a Metroid game given that both Metrod Prime 4 and Samus Returns are not beibg done by Retro. As for Donkey Kong, I can't find a source. I just remember hearing about it on a Q&A inreview shortly before Tropical Freeze was released. The strange part is producer Kensuke Tanabe confirmed Retro's next game back in 2014, but now has no idea what they're working on.

We will most likely see something next E3. Expect to see more on Metroid Prime 4, Animal Crossing, Pikmin, Smash Brothers, Pokemon, etc leading into next E3 and then they will throw the Retro title in as a surprise. 

Unless they plan a surprise launch for the game next spring to cause an industry stir, there is absolutely no reason to show it prior to next E3 with all the titles and Switch demand currently in place.