Well, to be fair, I've had people get annoyed with me for saying Merry Christmas in the past (I'm not particularly religious, but I was raised to say it and it's a habit to break)

It's all subjective. Just because you haven't had something said to you, or even haven't seen or heard of someone saying it, doesn't mean it doesn't happen.