I made that mistake a couple of times. I know most guides say that worms can eat a lot, but I found another that stated worms can go without fresh food for a week. So what I do is have a huge bucket and just put in a very small amount of everything and a tiny bit of dirt. Then once a week, I'll give them food scraps cut up, some leaves, and turn the pile with a spoon. I put a water bottle in with them to help with the temperature. I live in a desert as well so I water them once every three days ish but on one side, that way, if they feel like they have too much water, they can migrate.

One thing I did notice is once the pile got established, the pile itself got easier to self regulate. It seems the best way to get a compost pile started is to literally get some from someone else and start there. I wish more people around me would do composting so we could share.