I know that some body builds will stay bigger than others no matter how hard you exercise and burn fat. I believe that genes aren't the only reason for that.

because that's how bodies work?