The answer is that it is a birth defect. It really is that simple. As with any birth defect, special circumstances exist. That does not change the fact that gender is determined by what you were born with.
And yet... They still don't fall into your binary gender definitions. So what do you call them?
If someone was born with both a penis and a vagina, it does change their gender. Due to the birth defect, other things will need to be looked at on a biological level to determine which of the two genders they really are. Estrogen levels vs testosterone as one example. However that is an entirely different subject all together. The point is that just because there is a birth defect and other things need to be looked at to determine their gender, that does not mean that there are more than two genders.
The fact that it's a deviation from the normal binary "norm" in regards to gender is the entire point I am trying to make, that there is in-fact more than two genders... The amount of people that sit outside your pigeon hole definitions is entirely a redundant point, just one person who isn't conforming to your gender definitions is enough to break down your position.
Once you start talking hormones (Esterogen/Testosterone) or Chromosomes, then the puddle becomes even more muddy, there are men with high levels of natural Estrogen with female chromosomes, yet male appearance and reproductive organs.
Sometimes they have a complete male appearance, high levels of estrogen, female chromosomes and female reproductive organs.
It's not all black and white.
At the end of the day, you can only assume gender by ones outward appearance, you aren't going to know what reproductive organs they posses, you will not know their estrogen/testosterone levels, you will not know their chromosomes... Nor are you entitled to that information anyway.
So you might as well call an individual by what they appear to be and stop making a big production about it all.