First, the nature of human sexuality means that men generally "pursue" mates or try to impress them through some type of achievement while women typically attempt to make themselves "pursuit-worthy" by looking good. Women's bodies are a key aspect of this, with permanently enlarged breasts developed solely for a woman to promote that she is good reproductive material. As such, both men and women are far more likely to look at and evaluate a woman's body than a man's.
Second, as a result of the first point, women's fashion is far more developed and varied than men's, with clothing designed to either exaggerate or de-emphasise sexual features.
So because of human nature, and because women have a wide variety of clothing options, people evaluate and care what they wear.
