this post was submitted on 03 Sep 2026
186 points (99.5% liked)
Programming
28431 readers
136 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
founded 3 years ago
MODERATORS
The word is “scrapers,” as in to scrape.
no disassemble!
Well given the waves of "buy book, scan, scrap" scrappers might become an acceptable version as well soon.
load more comments
(2 replies)
load more comments
(10 replies)
did microsoft just lock up a good chunk of open source behind a (stochastic (for now)) login wall?
this feels like it should violate the gpl, but i bet it doesn't. truly devious.
Public GitHub repositories remain public and can still be accessed without a GitHub account, including repositories owned by paying customers. However, a subset of unauthenticated clone or fetch requests may now be asked to authenticate as part of GitHub’s protections against abusive traffic. If you receive a 401, update your application or script to use GitHub credentials.
Uh okay.
I wonder why people still keep up with this bullshit and not switch to some better public Git hosting provider.
Could you suggest an alternative?
I know Codeberg exists, but they had reliability problems recently IIRC?
sr.ht and Gitlab are paid products.
Technically one can self-host a forge, but my attempts at setting up CI were unsuccessful (IMO that's way more complicated that setting up the forge itself).
Codeberg's couldn't be much worse than github's reliability since 2019
load more comments
(6 replies)
you can self host gitlab too, and its free. yes, you CAN buy a license, but you can run it free forever. you can also use their SaaS free forever too.
at least right now, I think it's the best alternative, though I completely understand people wanting to favor OSS.
disclaimer: I have contributed code to gitlab, but I am NOT an employee.
Their SaaS is pretty good but of course you are running the same chain trust. You're betting that gitlab doesn't enshittify within the next 5 years which is hardly a guarantee.
Self-hosting gitlab is very resource-intense and complex from what I tried, though I did only try it two or three times.
I did set up forgejo which was way easier and less heavy but I haven't tested it much so who knows.
load more comments
(1 replies)
but they had reliability problems recently
Like GitHub, yes. But if you're not going to selfhost Forgejo they're the best option.
load more comments
(3 replies)
load more comments
(1 replies)
Our aim is to make public repositories accessible without authentication as much as possible. However,...
It's new reddit then. I was still using Github as a shitty backup for my projects, but an alternative may be required faster than expected.
Holy crap this is huge!
Open source is no longer open source on github.
Be ready to supply your fingerprints using Microsoft Authenticator app and your free source access subscription for only 49 USD per month! /s
This might be a problem for many projects. Rust, comes to mind, that pulls everything from Github and is 100% dependent on it. Same as Go. They pull everything from Github.
It might make sense for them to use something like radicle, a distributed git. Many people can easily pick what they want to distribute. Scripts can be written that make the local node only host the projects you depend on. That would naturally make popular projects more available.
Flathub
Nixos
GrapheneOS
uBlue (Bazzite, Bluefin etc), Secureblue
90% of android and linux apps
github is scary big
I'm worried about nix, honestly. An absolute truckload of nix packages pull their source from GitHub. It's the primary way to distribute flakes, too.
nixpkgs is literally a GitHub repo
load more comments
(2 replies)
I deleted my repos on github and moved to a mix of codeberg and selfhosted forgejo.
But I do occasionally fork/clone/reupload random projects from github to github so they have to spend money on storage and the scrapers.
Embrace. Extend. Extinguish.
Time to leave github! Boycott is the only language companies understand!
load more comments
(1 replies)
AI bot problem is real and there is no good solution to it. Look, I very much dislike GitHub for various reasons BUT currently there is no good way to throttle AI bots that literally trash web. They are like that geeky classmate who can never hold his liquors: it's nice having them around for some answers, but they ramble a lot and shit/puke in random places of the house making it unlivable.
have you never heard of anubis? tons of websites use that to screen out most of the AI scrapers.
They are the ones who created this crap
BUT currently there is no good way to throttle AI bots that literally trash web.
Sure is. There are plenty of protocols resilient to DDOS.
load more comments
(1 replies)
I believe that it is some fuckup and they don't know where exactly problem is and while they are looking for a way to fix it they've created plausible lie. When they'll fix it or believe that it's fixed there would be a public announcement like 'we heard the community and reversed our decision' and users will be happy. There is a serious need for github mirror, they are becoming less and less stable every year.
I'm 100% sure this is damage control on their part, they refused to acknowledge the incident and are looking for their way out.
What users found in this thread is
- problem is limited to EU
- problem is limited to subset of
gitbuilds (gix version x TLS lib x TLS lib version in place) - problem goes away if you switch back to HTTP/1.1 for some reason
If these are LLM scrapper mitigation steps then apparently fighting LLM scrappers is 7D chess game or something 🤷
The Microslop marketing department was always their best department.
Our aim is to make public repositories accessible without authentication as much as possible. However, like much of the Internet, we continue to see significant increases in the volume of robot traffic recently which has increased the need for verification, for example CAPTCHAs.
As I expected, it's about combating (excessive) bot traffic.
Hmm... I don't know whose building all those bots to scrape all of internet
"We're all trying to find the scraping LLM who did this!"
view more: next ›