i think there's a much broader discussion to be had here.
we can't ban racist people from existing, stopping a racist person from using a word, any word, in a derogatory way is not magically going to make this person less racist. if the intent behind using such words was to convey a racist or demeaning thought, then it's the intent that matters, nothing else.
i think if it truly was an accident and in a private conversation, then there's no reason for the guy to be banned. i'm more scared about thought police than people having racist thoughts. i'm pretty sure i've had them sometimes in the past and hope my better judgement helped me realize later on it was irrational to have those. in the same way i'm pretty sure plenty of people had racist thoughts about me and that's ok. it's part of human nature, one that we should continuously strive to be better about. hiding it, or pretending it doesn't exist will never solve the problem, instead will make racist people even more defensive about their "us" vs "them" mentality.