Well, I like them small.
It really depends on the personal taste. Problems do only exist if the woman with the small breasts always talsk about how small they are and how great it´d be if they where bigger. How can you explain you don´t care about her breasts size if she doesn´t believe you whatever you say?
And all breasts really count for are one night stands or short relationships. As soon as a guy has fallen in love with a woman (and I mean "with the woman", not her body) it doesn´t matter for him anymore. As a man you can see the womans breasts size by just looking at them for a second. After this second a man knows your size and there is no problem for him with it if he really likes you and not your body.
I think it´s legit when men talk about their size and if it matters, as a woman can´t see how big his little friend is. As a man you know what you get when you look at a woman.
And there are hundreds of things that are more important than breasts and body. there are things no other woman than my girlfriend can give me, no matter what their breasts look like. There is a difference between sexual preferences and love and friendship.
Personally, a good conversation with my girlfriend comes way ahead of sex. We can have fun and we like kidding us, we listen to music together, read the same books, are interested in videogaming (but just as a hobby. We play two or three times a week, not more), like to go dancing and having some party and I could do all this things together with her if she didn´t have any breasts at all.
So yes and no. They matter for sexual preferences and one night stands but as soon as you know "this is my woman" you don´t care as a man anymore. At least if you´re grown up to a certain degree.