There are levels of tolerance in there. E.g. I'm not gay. I have no interest in men. The idea of being sexual with a man is mildly repulsive to me.

With this, the bare minimum of tolerance is not actively working against the existence and legality of being gay.

Next is the "none of my business" level of tolerance. What happens between 2 consenting adults is down to them.

Above that is acceptance. Gay people have developed their own culture and community. While it's not for me, I recognise that its existence and celebration makes our overall culture more dynamic and interesting. It also provides a lot of happiness to others. Accepting and rolling with that provides a lot of positivity to others, without significant cost to me.

However, if I was approached by a gay guy and propositioned, there is no issue with me turning them down. I try and be polite about it, but being firm isn't being intolerant. (Luckily, most gay guys take being rejected a LOT better than some straight guys do).

Going back to your example. Going up to a black guy and expressing that, while you tolerate them not being a slave, you don't agree with it. This is intolerant, it is an incredibly strong dog whistle of your tolerance is forced.

Conversely, if, during a debate on religion and it's effects, you express your view that you accept people are religious, but don't agree with it, that is better. The context is a debate, and you can explain your reasoning better. It also lacks the dog whistle element that makes it bigoted.

Basically, context matters, A LOT.