this post was submitted on 30 Aug 2026
69 points (98.6% liked)
Opensource
6660 readers
719 users here now
A community for discussion about open source software! Ask questions, share knowledge, share news, or post interesting stuff related to it!
⠀
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
They're not scraping to cache or store, they're operating as an agent - scraping or single user requests.
Which is obviously bad and damaging, especially on their scale and on repeatedly fetched websites that they could be caching.
Google indexed the entire web. It's baffling that such indexing is not the norm on these huge providers.
I considered this, but would agents really ask for single commits with such frequency? They tend to get individual files via HTML, or do a git clone if they needed commit history for some reason.
Simple agents won't understand git. They can make web requests. If they want to check for reference code/source code, I assume they would request the rendered html.
I assume the frequency comes from many people asking various things, and the agents in the background pulling this data.
I'm not sure whether the scale-to-load ratio is plausible, because I lack the numbers, but it doesn't seem implausible to me that various agents for various prompts and tasks from various providers for many people repeatedly make these requests to a degree that significantly impacts the hoster.