Probably because the only paedophiles we hear about are those that do act upon it. So the word has become synonymous with molestor and rapist. Those struggling with their urges are hardly going to put themselves into the spotlight and hope they aren't judged and abused for it.
I don't believe that attraction to those younger (or older for that matter) is congenital (which I do believe is the case for sexual gender orientation). But instead caused by environmental factors.