Because what we experience as colors aren't provided by the physical world. The physical world solely provides a light frequency. We sense that frequency, process it, and understand it as colors because evolution has resulted in that being the most advantageous for us. It's better for us to understand what we see as colors than for example a numerical representation of the light frequency. It's quicker to process what we see as a color than a shit ton of numbers.

Imagine a painting where instead of colors, what you see is one shade with the numerical frequencies of all the light reflecting off of it. Colors are made up in our minds. It would be nearly impossible to make sense of that.

Colors are thoughts that exist solely in our visual cortex, an entirely made up human experience. For example, magenta doesnt exist in nature. It's what we make up in our heads when indigo and red are hit our eyes are the same time. Asking if another being sees different colors is like asking if another being thinks differently.