Cis man here. You're of course allowed to say breast and to talk about your chest as breasts, but it will be jarring to almost any other man. It's even avoided in places it used to be used.
When I was a kid in the eighties and early nineties the front pocket on a men's shirt was pretty exclusively called a breast pocket, we giggled about it but knew that's what it was, now I get funny looks if I forget it's a front shirt pocket.