this post was submitted on 09 Oct 2025
47 points (100.0% liked)
Programming
23053 readers
117 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Idempotence / self-healing: the system should be built in such a way that it tries to reach the correct end state, even if the current state is wrong. For instance, every time our system gets an update, it will re-evaluate the calculation from first principles, instead of doing a diff based on what was there before. This prevents bad data from snowballing and becoming a catastrophe.
Giving yourself knobs to twiddle in production: at work we have ways of triggering functionality in the system on request. Basically calling a method directly on the running process. This is so, so useful in prod issues, especially when combined with the above. We can basically tell the system "reprocess this action/command/message" at any time and it will do it again from first principles.
Debugging: I always first try and find a way to replicate it quickly. Then, I try and simplify it one tiny step at a time until it's small enough I can understand in one go. I never combine multiple steps per re-run and always verify whether the bug is there or not at every single stage. This can be quite a slow approach but it also means I am always making progress towards finding the answer, instead of coming up with theories which are often wrong, and getting lost in the process.
Would you be willing to give an example of the second? I feel like my boss would throw a shitfit if I told him I wrote anything that even remotely alter prod
Certainly! The line we don't cross is that we don't directly edit data. Every record in our database must be generated by the system itself. But, we can re-trigger behaviour, or select different flows, or tweak properties around the edges as much as we want.
For example: