I don't know if my perspective matters here, but...
The only question is this: are you willing to take the risk of the guy trying to do something stupid, or doing something that could potentially hurt your girlfriend or - God forbid - your daughter in some way?
I mean, if I had to choose between the well-being of my brother and the potential well-being and security of my wife... shit, man.