Our idea of "warm" colors is just a historical accident. Wood burns orange because it produces soot, which glows orange in the flame. Same applies to candles too, because there’s never enough oxygen to burn all the carbon.
If we’d started with purified hydrocarbons instead, blue might have been the ultimate "warm" color. Natural gas burns with a blue or even invisible flame, a sign of complete combustion. Orange, then, would be the color of flawed, struggling fire.
Imagine a house heated and lit by a gas furnace instead of a traditional fireplace. The light from the fire would be blue, and we’d associate that glow with warmth and coziness. Picture old paintings with a cozy atmosphere, their hearths glowing blue. If everyone grew up like that, blue would be the warmest color instead of orange.
"Different flame types of a Bunsen burner depend on oxygen supply. On the left a rich fuel with no premixed oxygen produces a yellow sooty diffusion flame; on the right a lean fully oxygen premixed flame produces no soot and the flame color is produced by molecular radicals, especially CH and C2 band emission."
Source: Wikipedia
I'm going to add to the sun point, and remark that the cool range of tones is associated with snow and ice, since these reflect the blue of the sky. So even if people didn't need fire to cook or live their lives at all, they would associate the blue ranges with winter.
Water in its liquid state also looks blue, and is usually associated with something refreshing in warm areas. So even in the tropics you would have a reason for associating blues with cools.
In contrast, most desert areas are naturally in orange ranges. You can also argue autumn forests would look orange and they're not exactly warm, but the light in an autumn forest doesn't bounce as much or as tinged compared to light conditions in desert areas, where the soil or sand is yellow or orange -red, and where whatever casting a shadow is probably the ground itself.
Finally, human bodies are more in a warm range tone (no matter the skin color) when compared to the same body suffering cold. Nails and lips take on a bluish hue, so do fingertips.
I think you would need to tweak your universe a little bit more to achieve a reversal in color association.
Those are some pretty solid points! Especially the one about healthy vs. freezing skin tone.
Those substances are blue due to Rayleigh scattering, not because they reflect the sky
The highest energy (shortest) wavelengths scatter first, so out of visible light blue is the first scattered colour