Hmm, good question. Horror films are all over the map - lots of them use cheap camera tricks to scare the heck out of you -- sudden screams, knives flashing, a sudden gleam of fangs. None of those tricks would work with videogames - an instant kill event is just an annoyance, nothing more ("oh dear, my character just got decapitated... mustn't do that again").
Neil Manke's "They Hunger" mod had some truly spine-chilling moments, while Silent Hill 3 seriously creeped me out. Those walls oozing blood... still gives me the willies.