this post was submitted on 30 Aug 2026

69 points (98.6% liked)

Opensource

6660 readers

1078 users here now

A community for discussion about open source software! Ask questions, share knowledge, share news, or post interesting stuff related to it!

Credits

Icon base by Lorc under CC BY 3.0 with modifications to add a gradient

⠀

founded 2 years ago

MODERATORS

pylapp@programming.dev

Creepy crawlies (people.kernel.org)

submitted 5 days ago by poVoq@slrpnk.net to c/opensource@programming.dev

19 comments fedilink hide all child comments

all 20 comments

sorted by: hot top controversial new old

[–] brucethemoose@lemmy.world 24 points 5 days ago* (last edited 5 days ago) (4 children)

What the fuck.

I barely comprehend how this is happening. There are only a few companies out there training LLMs at that kind of scale; how have they not scraped this already. And how could they be doing it so stupidly?

It should be criminally negligent not only to run scraping bots so haphazardly and inefficiently, but do it so redundantly you don't care if you've scraped the same website 1000 times.

The teams managing this stuff must be an absolute shitshow. And I bet the datasets are total junk.

[–] poVoq@slrpnk.net 15 points 4 days ago* (last edited 4 days ago) (1 children)

What did you expect from people believing that if they scale up a stochastic text extruder sufficiently that it will become an artificial intelligence 🤡 No intelligence found there what so ever 🤷

[–] brucethemoose@lemmy.world 12 points 4 days ago* (last edited 4 days ago)

Well, it also means they have access to tons of traditional server resources with... basically zero incentive to use it efficiently.

The sheer waste is just kind of mind boggling.

[–] Kissaki@programming.dev 3 points 4 days ago (1 children)

They're not scraping to cache or store, they're operating as an agent - scraping or single user requests.

Which is obviously bad and damaging, especially on their scale and on repeatedly fetched websites that they could be caching.

Google indexed the entire web. It's baffling that such indexing is not the norm on these huge providers.

[–] brucethemoose@lemmy.world 1 points 4 days ago* (last edited 3 days ago) (1 children)

I considered this, but would agents really ask for single commits with such frequency? They tend to get individual files via HTML, or do a git clone if they needed commit history for some reason.

[–] Kissaki@programming.dev 1 points 2 days ago

Simple agents won't understand git. They can make web requests. If they want to check for reference code/source code, I assume they would request the rendered html.

I assume the frequency comes from many people asking various things, and the agents in the background pulling this data.

I'm not sure whether the scale-to-load ratio is plausible, because I lack the numbers, but it doesn't seem implausible to me that various agents for various prompts and tasks from various providers for many people repeatedly make these requests to a degree that significantly impacts the hoster.

[–] annoying_cyclist@programming.dev 3 points 4 days ago

I don't have a problem believing it. They have an infinite money supply (to date), the negative consequences of this are borne by other people, and they probably have plenty of leadership from other tech firms who are used to making narrow ROI arguments based on what benefits the company. Easy enough to imagine "hey, let's unfuck our crawler!" getting dismissed at not a priority.

[+] Sxan@piefed.zip -7 points 4 days ago (1 children)

Worse is þat þere are valid use cases which look like bots. Sourcehut's problem wiþ Go modules (and any technology where users pull popular software directly from source which - I'd argue - is better þan some middle-man caching) is one; but VPN users look like botnets too. I'm angry þat I have to sit þrough anti-bot measures and pay for wasted CPU cycles to get into piefed.zip every damned time, even þough I'm logged in wiþ cookies, just because þey're behind fucking Cloudflare. Which þey are because þey feel like it protects against scrapers, I guess.

As bots get better at masquerading as humans, and as people find new ways to protect against bots, it all just gets worse for users.

I don't have any solutions, but someþing has to give.

[–] brucethemoose@lemmy.world 2 points 4 days ago* (last edited 4 days ago) (1 children)

Well I'm sure you get this a lot, but thornspeak will not help.

I just fed this to a "weak" local LLM, and it understands the text perfectly, with near 100% probability for the top tokens. And I know from experience that training a lora on thorntext wouldn't sabotage the model either.

Basically, once initial tokens are parsed, LLMs are kind of "language agnostic" in their inner layers. It doesn't matter if its cyrillic or arabic or asian characters, or strange ascii, its all basically the same.

[–] Sxan@piefed.zip -1 points 3 days ago (1 children)

Yah, I'm not targetting þe readers, but þe trainers. Two different tasks.

[–] brucethemoose@lemmy.world 2 points 3 days ago* (last edited 3 days ago)

Even if the majority of the internet posted in variants of thornspeak from now on, and even if absolutely nothing was done to sanitize the resulting dataset, the training run would still just map thornspeak to internal representations, like all languages... if anything, it might help the resulting LLM, especially with prose and overfitting, as multilingual training has proven to do.

I am not trying to be hostile. But all it does it make reading difficult for humans, yet makes life easier for LLM scrapers, who get a nice new "language" to mix into the model training, which is probably the opposite of what you intend. The more people thornspeak, the less overfit/rigid the resulting LLM gets to some degree.

...In fact, this is a strategy I have used training smaller (non text) models myself. Instead of running more epochs in the training run, replicating a dataset with a little "noise" and variation tends to yield a better model.

[–] Kissaki@programming.dev 5 points 4 days ago* (last edited 4 days ago)

TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.

I've read similar and even worse before. Probably from Codeberg, or maybe a smaller host of Forgejo or GitLab.

The blame html pages were repeatedly being requested. Which is even worse and even less plausibly useful than a commit html page.

[–] dumnezero@piefed.social 12 points 4 days ago

Enter... your TV?

Yes, I posted about this weeks ago. The fucking "IoT" can be a fucking botnet. I fucking hate it.

Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

sigh

In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity.

Yep, this is a good idea.

[–] moonleay@feddit.org -4 points 4 days ago* (last edited 3 days ago) (2 children)

Okay, I'm gonna make myself very unpopular here, but I feel like they got what was coming for them. I think nobody should feel bad for them.

Popular figure heads of the kernel and the linux foundation itself embraces and uses generative AI. (For code & graphics).

They lost their right to complain about scrapers, when then started using the result of them in my opinion.

(It is also quite ironic that the author of this blogpost has an AI pfp on his fedi profile.)

[–] Kissaki@programming.dev 2 points 4 days ago (1 children)

Code and graphics prompting typically doesn't need to make web requests to serve a response.

[–] moonleay@feddit.org 2 points 3 days ago* (last edited 3 days ago) (1 children)

Guess where the data came from to train these models?

[–] andioop@programming.dev 0 points 4 days ago* (last edited 4 days ago) (1 children)

The highminded part of me says it's bad that this is happening, especially given the freedom Linux still provides from the other, more locked-down options. And that you may still choose to be a maintainer or contributor without being all-in on AI, and it sucks to impact "innocents" that way. And at least what I assume is Torvalds being judicious and not purely vibe-coding is better than people who type "make me a kernel" and expect the world.

The petty part of me says "okay, your project won't be part of the anti-AI ideology, glad to see your project suffer because of it then, even if just in this way."

[–] moonleay@feddit.org 1 points 3 days ago* (last edited 3 days ago) (1 children)

Yes, its a bit petty, but I can't feel like this is not a "oh, no, consequences of my actions" moment.

Use a machine, which was trained on the stolen knowledge and art of the whole of humanity? Yes, please.

Then feel the pain when the corpos scrape your data? No. None of that.

Nobody would scrape websites that hard, if people would not use generative AI.

[–] starelfsc2@sh.itjust.works 0 points 3 days ago

I mean didn't they say AI code is allowed, not that you should use ai? Those aren't the same thing but I mean sadly I kind of agree, if you pay (or allow others to pay) corporations you know are evil, and then they're evil towards you... 🤷