Life isn't all roses. It sounds like you have found sucess. So, what are you complaining about? Be happy! Not everyone can do that.

Did you ever had severe anxiety? :p

It isnt always solved as easy as one might think.

You said it made it, "worse." So it was there beforehand. (Obviously, I know) I tell people this all the time who suffer from depression or social anxiety since the two go hand in hand, there must be an underlying issue that is the root cause for your condition. Much like your fame, other factors have contributed to your ailment but aren't necessarily WHY you're suffering.

You need to uncover what exactly it is that causes the anxiety. Is/was it a person? Is it people in general? Was it an event? More than likely it is a present occurrence since it's an ongoing issue. Anyway find out what it is an ACCEPT IT. Accept being afraid of it. Then, once you've realized you're afraid of whatever it is, ATTACK. That's when you address it in a serious manner..

You are a good person and you're doing a great thing, the only ones that should feel bad are the ones that insult, harass or threaten you. If you can't ignore them, at least never forget that what you're doing is worth infinitely more than what they're trying to do to you, wasting precious time they could spend far better, by the way.