My grandmother who raised me always did the "I don't have any money on me" thing. And I always followed suit until pretty recently. But I got to thinking more about it and eventually concluded that I should always keep a $20 or two in my pocket ready to give.
And then the pandemic came along and I didn't go out much. And now I work from home full time and don't often go anywhere that I'm likely to run into folks asking for money. But I have put that into practice a few times and felt good about it.
There was a woman with a sign standing outside the post office. I ignored her on the way in with the intention of giving her a $20 on the way out. And I made good on that intention. It was scary, but only because I'm kindof an agoraphobe. Heh.
I do have the means to go handing out $20s willy-nilly. And of course with how infrequently I'm likely to pass folks asking for money out in the world, the rate at which I give is tiny. But I do give when that situation comes up.
I'm not saying you should give $20s out to folks. But if your financial situation is stable, I'd say you should give what you can in those situations.
And the fact that these thoughts/questions/concerns are rattling around in your mind are probably a sign of personal growth, so good on you for that.