Two tips that work for me:

• After cargo add I have to sometimes run the "restart rust-analyzer" command from the vscode command pallette (exact wording may be off, I'm on my phone as of writing this comment). Much faster than cargo build.
• Consider using sccache to speed up rebuilds. It helps a lot, though uses a bit of disk space. But disk space is cheap nowadays (as long as you aren't stuck with a laptop with soldered SSD, in which case you know what not to buy next time).