Which games for you stand out the most as being unlike what you'd expect from a Nintendo developed/published title, and in what way?

There were a few exclusives on WIi that I was surprised by, not sure if Nintendo published them but one black and white game with chainsaws and red blood stands out, A game called Mad World. It was Platinum who made it but it being on the WII at all set a very different tone for Nintendo consoles going forward. Esspecially as an exclusive.

Mad World was the game that got me to borrow a Wii from a remote familiy member ("don't you want the balance board as well?" - "no, just the console"; "how about Just Dance?" - "I'm good, thanks") and rent the game at the local Blockbuster. It would have stood out even on the other consoles, but it was especially notable on the Wii. I would say Conker's Bad Fur Day on N64 was a similar case.

Was Dinjin the Giant developed by Nintendo? If so, then yeah that’d definitely be up there. That alongside that one military-strategy microphone game Nintendo made at the very end of GCN. Captain Rainbow would be another one. If looking for something a bit more recent, I’d also say Emio and Endless Ocean.

At one point in time, I feel like Xenoblade Chronicles could have been one of those games, though with how much the franchise has grown and how closely associated with Nintendo it already is, I’m not sure if this is the best answer.

Any M-rated game published by Nintendo. It's quite rare.
Eternal Darkness, Emio, etc.
Cadence of Hyrule: Published by Nintendo outside of Japan and has the Zelda IP. It's also surprising anytime Nintendo published Nintendo Warriors games, especially starting with Hyrule Warriors (again, published by Nintendo outside of Japan).

I was pretty surprised by the next big Miyazaki FromSoft game is being published by Nintendo.

Sin & Punishment games. Nintendo IP. Made by Treasure. These are games that feel very SEGA, not Nintendo and yet they beat SEGA as the very genre they invented.

Wii U era is an odd one. Nintendo wanted to hang with the cool kids so they published Ninja Gaiden 3 Razors Edge. Bayonetta 2. Devil's Third. 

Another one on Wii but Japan only is called Captain Rainbow. In the game there is a mission to find Birdo's vibrator under her bed to prove she is female. Yes Birdo from Mario masturbates.

Nintendo never published Bad Fur Day. Just Rare. Nintendo did not Publish Madworld either. That was SEGA.

Geist on the GameCube (developed by n-Space). M-rated first person shooter / adventure with some possession mechanics. It was quite fun (although I never completed it).

Arms, not because i dont think Nintendo wouldnt publish a game like that, but rather because it was really not good.