Wikipedia

People complain about yellow paint but not a HUD because the HUD is not part of the game world. It's an abstract representation of state to the player but explicitly not something any character in the game sees.

It's like the score in a movie. The music is (usually) not playing where the characters are but its presence is understood as a representation of how the characters feel. It's an out-of-band message.

Yellow paint is part of the game's world; it's an in-band message. Someone put that paint on things and somehow only on things that are breakable or climbable. That strains some people's suspension of disbelief – it's more plausible that someone created a zombie virus than that someone went through town and color-coded everything by degree of interactivity.

In-band messaging in games can be done subtly but that requires a very competent designer.

It's easier with less realistic visuals and more limited means of interaction; since everything is abstracted anyway, indicating interactivity becomes easier – e.g. breakable walls have large cracks and one early on had to be broken to progress. Now the player knows how to spot and break such walls.

But when you have many dissimilar objects with many dissimilar forms of interaction in a realistic environment and you don't want to show abstract prompts for the sake of greater immersion, you need to indicate interactivity in some other way.

So you break out the yellow paint and break immersion for some players.