Retro does an amazing job with the Metroid series
you seem to really be reading into the eeiry vibe of Super Metroid, but do recall that first and foremost Samus is a bounty hunter. not all of her adventures are going to be insanely creepy in the sense that you desire
also part of the reason Super Metroid has that certain dark vibe is because it has to- the SNES was only capable of so much, a lot of what I think you are connecting to with is the music and isolated feeling (like you say, like Alien)
I think Metroid Prime got it exactly right. its creepy but still pretty at times and intriguing. great controls, great weapons, great mysterious space vibe
I think others have mentioned already, but I agree with some that part of the vibe you really enjoyed was BECAUSE it was a 2D sidescroller. it is very difficult to create a similar atmosphere in 3D
