What would you do in my situation?

[dsister44]

Just tell him that if he stays at your house he has to be out of the house the same times you are. And if he is alone in the house with your GF then you will kick him out. We did basically the same routine with my brother's friend and my sister