Fuck AI

I don't know that your solutions are viable for a lot of companies, but I can believe that it is effective. It appears your situation has a lot more room for what I'll call "decisive choices" that ours. Our company 100% wants to be indexed on every search it can, so blocking most of the official crawlers is out, though I do limit access to only the places we want them to index for anything I can identify as a bot or bot adjacent. As for the vast majority with the bullshit user-agents, historically, heads tend to roll here when blocking content/requests for legitimate users, so while false positives happen, they need to be kept to a minimum. I've had to roll back several mechanisms that somehow ran afoul of edge case users. So I'm open to trying something similar, I won't be able to do it like you are and will probably have far less success as a result. Not that you don't likely already know this, but I believe that if your solution does become more mainstream, many crawlers would probably adapt to simply make more thorough use of sec-fetch-mode and other headers to more believable as a valid client request.