If it upsets someone to hear you say it, don't say it around them.
It's so simple, and yet so many people overcomplicate it. If you're around company, and you don't know if saying it will upset someone, don't fucking say it. Black, white, red, or yellow. Don't say the word. If you're around friends who are all cool with it, go ahead and say what you want. Black, white, red, or yellow. It's not the word that matters - it's the audience who hears it. Know your audience.
But there absolutely is a double standard. If blacks want to reclaim control of a word that was used to control and oppress them for hundreds of years, let them.
I'm African American, and hearing white people throw around the word rubs me the wrong way. None of my friends say it around me, because they know it bothers me. I know some of them say it with their other friends, and it's fine. I'm not there to control what they do when I'm not around. Sometimes they even slip up and say it in front of me, and I'm always lenient because they always express remorse. Again, it's all about knowing and respecting your audience. Not just about the "N" word, but about anything and everything.