What's crazy is that they aren't just doing this because they make more money with AI.
No, they really are making more money by selling whole wafers rather than packaging and soldering onto DIMMs. The AI companies are throwing so much money at this that it's just much more profitable for the memory companies to sell directly to them.
Sure, and if the AI companies want to configure their crawlers to actually use APIs and ActivityPub to efficiently scrape that data, great. Problem is that there's been crawlers that have done things very inefficiently (whether by malice, ignorance, or misconfiguration) and scrape the HTML of sites repeatedly, driving up some hosting costs and effectively DOSing some of the sites.
If you put Honeypot URLs in the mix and keep out polite bots with robots.txt and keep out humans by hiding those links, you can serve poisoned responses only to the URLs that nobody should be visiting and not worry too much about collateral damage to legitimate visitors.