My hands are pretty small for a male as well. I have made that observation at some point, but it's not something I think about constantly. I don't know if it has been a factor when dating and meeting meeting women or not. It may have been the case that I' have been denied second dates, because I've heard that a woman looks at a man's hands, but I wouldn't know. I have never heard comments about them from anyone, male or female. I have had many long relationships in my life, and was even married at one point, so it has not really been a hindrance for me.
Things of this nature tend to be bigger deals for oneself than to others, so I say don't worry about it. Don't let yourself be your biggest enemy about this. You're not alone in this, anyways. Everyone you meet will have something about their body that they are insecure about, so it kinda levels the playing field.