Is "Bugger" a swear word???

[Aj_habfan]

Only time I ever hear people say is if that say something like, "Look at the little bugger" - and it is actually refering to a smaller person. The word alone doesn't seem bad at all.