Not only is the word fag used by a lot of people, because there are a lot of hateful bigots out there, but even when you don't mean the nasty implications, it still reminds gay people around you how much the world hates them and leads hateful bigots who overhear you to believe that their views are more widely held and acceptable to share in public. Shocking though it may seem, South Park is not a moral authority on these matters.

Aside from that, if you know a word is commonly used a slur against a disprivileged group, someone advises you to stop using it, and your response is that you'd rather say it, hurt someone and apologise if they complain about it than just stop using that word, what does that say about your priorities?