Complicated, man. Human evolution has basically been about making our adults look more and more like children, specially women. We favour younger-looking features, a trait called neoteny.
See http://en.wikipedia.org/wiki/Neoteny for some clarification.
Attraction for children seems like a rather extreme form of that trait, one that clashes right on our instincts of nurturing and protecting our children and societal attitudes towards sex. No wonder its taboo, and always will be (at least I hope so). By the way, we did the same thing to dogs over the last few thousand years. Dogs are basically wolf pups, like we are almost hominid children.
Edit - it's actually striking when you stop to look how much an attractive women can look like a child. This is fairly unique among mammals, which only goes on to credit the importance of neoteny on our evolution:
