You get used to it, it just takes time. You can also ask your prescriber for something to help with sleep. I take trazadone. There's also nothing stopping you from taking it earlier.
You might try to switch up meds too. Adderall felt very "speedy" for me, even the extended release. It releases in 2 waves 4 hours apart.
I much prefer concerta which is the same med just with a better release mechanism (steady all day). In my experience ADHD meds are like antidepressants you might have to cycle through a few to find the right fit
