Push the stick into any direction, get real close to it and blow out all the dust you can on the side opposite to where you pushed it. Then repeat this in several directions, I usually do 3 or 4. Then spin the stick around like a dozen times, and another dozen times the other way.
I know that sounds stupid but in my experience it does temporarily fix the drifting.