Most people get choosy about their inner circle as they get older.

Yes, making new friends is hard. Wanna know what else is hard? Living life surrounded by fake friends that make you not want to continue in life. Choose your 'hard'.

The issue is that most of us still have to interact with society at large outside of our inner circle

What part of that means you have to put up with disrespectful behaviour?

and no, telling people who you interact with at work or while grocery shopping to fuck off doesnt end well

The 'fuck off' is not meant to be literal. The point is to stand up for yourself. If that isn't viable, cut them off.