People who live near Spanish speakers and actually bother to ask about it won't be offended. But I'm talking about like random white people who do unironically think it's a slur. From my understanding talking with a Mexican coworker years ago, he told me it's basically neutral. It can be said in a good way, bad way, without meaning, but white people who don't know anything and have an inherent bias against Spanish speakers will interpret it as negative.