It’s not just a fat or muscle thing. Those both contribute of course; fat insulates and muscle produces more heat. But the real player is the surface area to volume ratio.

A bigger person has a lot more volume than they have a bigger surface area, and since heat is lost through the skin this has a major impact.

I don't even think it makes sense, since your skin senses it.

It makes sense when you learn that your skin doesn't sense ambient temperature at all, but rather it senses the rate at which you are losing or gaining body heat. This is why metal can feel cold at room temperature while something like a blanket actually feels room temp, it's a better heat conductor so it absorbs body heat from you faster.

Having more body mass means you produce more body heat at any given time, so the rate at which you lose body heat to the air is decreased, making you warmer.

Could it just be a weight thing? So not necessarily fat, muscle could also help.

If you're body is heating your whole body, the amount of heat added will increase linearly with volume. But your surface area, i.e. the skin, increases sublinearly with volume. So you get more heat per surface unit?