If you don't prematurely optimize your code you'll never optimize your code.
I know people are advised against premature optimization because it risks you wasting time on something that ends up unused.
But let's be honest, once you get something into a working state will you ever be afforded the time to go back to it? Even if you are, are you really gonna choose to uproot code and do a refactor on it and potentially everything relying on it?
Rather than waiting until your hacks and workarounds run out and you absolutely have to do a refactor, I would prefer spending more time upfront to make sure it is efficient and flexible before moving on.