Codename S.T.E.A.M. is one of the more interesting games Nintendo has done this generation. I don't say it's a great game or a bad game (I haven't played yet, though I'll get it at some point), but just everyones reactions to it really seem remarkable. In a time when everyone critizised Nintendo for not creating enough IPs, the gaming community responded to this game with an apathy I've not seen in a long time. Nobody knew about the game (blame Nintendo for the lack of push) and those who knew it had a lot of mixed reactions to it, resulting in one of the worst selling Nintendo games on the 3DS.

There were a lot of design choices that hurt the game, from mixing cartoony characters and story (alienating older audiences) with a really deep and complicated gameplay (alienating younger and more casual audiences), making the game around american folklore and characters (alienating it for those outside NA)... But a thing that we have to take into account is that, at its core, the gameplay and the concept is not bad. A lot of people said it would be a great game if Nintendo just took notes from Awakening and brought a more serious artstyle and story, even anime-esque, something similar to Valkyria Chronicles.

Could we see a new game based on the gameplay of CodeName S.T.E.A.M. on the NX? Maybe a different story, artstyle, tweaking the gameplay to make it work better on home consoles... Or will they be too scared because of S.T.E.A.M.'s flop to even try it?

Didn't the game flop? I think probably not, something like Kid Icarus NX would likely have far more commercial appeal if they want to bring back something from the 3DS-era.

Well, the idea I had is a Codename STEAM successor that only preserved the gameplay and changed everything else (story, artstyle, characters...). Whit the apathy the game had, it would be dumb not to make those changes.

Knowing Nintendo's history, I wouldn't be surprised if it is the NX's flagship title.

We could, but I doubt they will pick this one up for a while

Still waiting for a new Advance Wars game..

I'd say almost definitely not. I believe it's literally the worst selling Nintendo first party game since the Virtual Boy.

Not by a long shot.

http://www.vgchartz.com/game/24655/captain-rainbow/

I stand corrected. Not that I trust VGC's numbers slightly, considering by all metrics I've seen STEAM sold much worse than this site indicates. Either way one of these two games had a marketing budget and was supposed to sell. One...one I, as a Nintendo fan, have never heard of. 

Edit: Also they matched sales in Japan according to VGC and apparently Captain Rainbow is Japan only, so that doesn't exactly instill confidence in STEAM

I think Nintendo was planning an european release, but they scrapped it. Maybe it was the sales, or maybe because the game was so japanese. But it never came.

That's what I'm reading too. I also found out that Steam sold worse FW in Japan than Rainbow, and that Rainbow sold 20X as much lifetime as VGC shows. So it's closer than we thought, which makes things even sadder for Steam.