The shirts/pants are supposed to represent a person's experience/genetics in this analogy. You said you can hate a person's experience/genetics, so you can also hate the shirts/pants.
the combination of the shirts & pants is supposed to create the shirt-pants outfit. Much like the combination of experiences & genetics creates a person.