I played it from five years old through high school (among other sports) and enjoyed it, but I rarely met anyone who had aspirations to do anything professional with that, unlike other sports.
I'm really not sure why it didn't catch on as a spectator sport. Baseball has really old roots and was our first team sports league that resembles the modern leagues of today, so that had everyone's attention early. Otherwise, we really liked Rugby and Lacrosse early on, and American football seems to resemble those more than soccer (early American football was very much like rugby).
From there we received Basketball that caught on overnight which, along with hockey, became big as it cold be played in cold climates. The infrastructure, from college teams and professional leagues, definitely helped as equipment was always available, as was cheap equipment mass produced and affordable for children. Many of these sports would otherwise be difficult to play and organize.
I guess, then, it got lost in the noise, what with baseball, American football, hockey, basketball, and even now-less-popular sports like Lacrosse and Rugby.
Still, the idea that soccer is simply a game for children has definitely begun to dissipate here, and there's a growing interest in the sport. The main problem it seems to face now is that, while we enjoy watching our national team, the professional league has not been nearly so successful as gaining the support and attention of your average Americans as many of the others do... I know more people here in jacksonville that support our Arena football team than do our actual football team.
Athletes are beginning to consider that sport more than in the past though, and that Title IX has lead to a very large and talented women's community, but the entry barrier is certainly lower among women. The U.S. men are no longer complete pushovers, though, so I think things might be changing.