https://en.m.wikipedia.org/wiki/History_of_mathematics

"From 3000 BC the Mesopotamian states of Sumer, Akkad and Assyria, followed closely by Ancient Egypt and the Levantine state of Ebla began using arithmetic, algebra and geometry for purposes of taxation, commerce, trade and also in the patterns in nature, the field of astronomy and to record time and formulate calendars."

The first "true" pyramids were not built until ~2613. Prior to that it was all step pyramids, which are much less complex - just put a bunch of consecutively smaller squares in a stack. Even then, Djoser was started in ~2670, several hundred years after the "introduction" of basic math. Just because we don't have extant physical mathematical texts surviving from that time doesn't mean they didn't know how to do math.