Here's my experience (recently diagnosed ADHD, suspect maybe AuDHD):

Starting about 6 years ago, I started having fairly large amounts of caffeine (300-400mg/day), probably as a way of self medicating ADHD and compensating for too little sleep). At this point, I realized I was physically addicted to caffeine, and if any day went by where I didn't have caffeine by say 3pm, I'd get a bad headache. For the next ~3 years after starting this, I didn't go a single day without coffee.

Starting about 2-3 years ago, I decided that I really didn't like the idea of having to have a coffee every day, so I decided to try to stop my addiction. I stopped abruptly and didn't have any caffeine for a month (for me, I knew a gradual stopping wouldn't work, I had to fully commit to it to get it to stick). It was miserable, especially for the first 3-4 days. The first week as a whole was pretty rough, then week 2 was marginally better, week 3 a bit better still, and week 4 felt good, finally.

Since then, because I really like the taste of coffee, I've been consuming 2-3 coffees per week, but I try to avoid having them on consecutive days. I think it works pretty well for me, and I have managed to hold off the physical addiction like that (except for one time where I compensated for jetlag with lots of coffee, and got re-addicted and had to do the same miserable process as above).

Also, I recently started stimulant meds for my ADHD (Vyvanse in particular), and for a while I couldn't have caffeine because it made me too anxious. Now, I'm used to the meds and can have caffeine, but it's much easier to resist the temptation.

All in all, I'm happy that I'm not addicted anymore, and for me it was a painful process but worth it. I find that caffeine messes with my sleep even many hours after consumption, and I don't feel as rested on days when I have it. I've mostly switched to drinking tea or decaf coffee, and I think it's made me feel less anxious too.

Let me know if you have any questions!