Zelda: Majora's Mask...I think I have mental scarring from that game...why? because of this guy:

the terror...

People, people, please.
The Title says "...but you don't know why"

I came in looking for how this could possibly make any sense, and it's just a buncha nerds saying holy crap Doom 3 is so ska-weee. AAIIEEEEEE!

AD&D: Treasure of Tarmin (Intellivision): turning around a corner and suddenly fronting a wraith scared the crap out of me.

PC games:
Thief games (particularly the haunted cathedral and the catacombs and "Escape!"in Thief I and some missions in Thief II, I still have to finish T III, but the Abysmal Gale ship was moderately scary)
Hexen I: lots of scary levels
Heretic II: the first town
Ecstatica II (some locations)
Gothic (some locations)

Edit: if I must limit myself to "I don't know why", then, some strange atmospheres in Ecstatica II, Hexen I, Heretic II and Thief I,II,III

Ocarina of Time. Bottom of the Well and Shadow Temple freaked me out. :p

Seriously, this guy? :o

Those leaf faced kids in TLoZ: Wind Waker......I think they were called Karoks. Freaky!!

^ Koroks, actually. :p And what? They were cute as hell!

Most recently would be the Maw.

The game's colorful and rounded, the gameplay is easy and very forgiving, the story is simple. It's has all the earmarks of a nice sweet kid-friendly game, but late into it I realized the Maw consistently got bigger with each stage and was starting to get freaked out by the fact I'm nurturing a monster into maturity. I kept wondering if the people who made the game designed it like that intentionally. The ending just kind of freaked me out when it seems like that's may of what the developers were going for.

And also the atmosphere in the lost pueblo near San Andreas fault in Duke Nukem 3D was quite scary.