Programming

Much like... -reads notes- Microsoft is via Github... huh....

We want to give developers agency over their source code by letting them decide whether or not it should be used to develop and evaluate machine learning models.

crawled directly from GitHub and built to pre-train code LLMs with full-repository context

Repositories that opted out are removed from the dataset before each patch release.

"agency"

Which AI company will not use v1 which has all of the data but will use later patch releases instead which have less data?

I just followed the opt out link and they're making you open a public issue with a Markdown list with all your repositories you want removed.

Surely there has to be a better way to do this (probably the point).

Also why are they storing 4.71 TB in a Git repo? How are they going to deal with removal requests?

EDIT: It seems like they're manually responding to the issues, wtf? There's a perfectly fine GitHub auth system that they could use to verify everyone's GitHub account / repository ownership

EDIT 2: This is apperently a collaboration between Hugging Face and ServiceNow, why is my companies IT ticketing system scraping all of GitHub?

Noteworthy: They crawled only the default branch HEAD and inlined all source content.

• The file contents are included inline. The decoded UTF-8 source text is embedded directly in the dataset, so it is fully self-contained — you can start training the moment the download finishes.
• It reflects the state of GitHub in August 2025. The corpus is a direct crawl of GitHub repositories at their default-branch HEAD, capturing roughly two additional years of open-source code compared to The Stack v2.

When I hear this company's name I can't get the idea out of my head that it's related to the face huggers from Alien.

lol my first ever JavaScript project is in the stack. No wonder their models suck so much ass

From the linked webpage readme:

2.1 Classify. Each file is labelled permissive (at least one permissive license detected, no conflicting non-permissive license), no_license (no licenses detected, or only non-license legal texts such as CLAs), or non_permissive. The permissive allowlist follows the Blue Oak Council list plus licenses categorized as Permissive or Public Domain by ScanCode. Files classified as non_permissive are excluded from both released datasets.

From https://www.bigcode-project.org/docs/about/the-stack/:

v1.1: The three copyleft licenses (MPL/EPL/LGPL) were excluded and the list of permissive licenses extended to 193 licenses in total. The list of programming languages was increased from 30 to 358 languages. Also opt-out request submitted by 15.11.2022 were excluded from this ersion of the dataset. The resulting near-deduplicated dataset is 6TB in size.

So MPL/EPL/LGPL are already not part of the dataset.

So… why were they in there? Was this added for v1.1?

"one permissive license" - So if my project includes a lib and I include the license file for that for the license notice…?

I never really needed any justification, but ever since AI companies just take stuff illegally, and that openly, it has become my justification to just pirate the shit out of everything.

(Excluding indie games of developers I like).

Am I alone in not wanting to put my username into that field? If they don't have it will they then just decide that it's now a good time to scrape it? Or are they going to record that it was searched?

they can do whatever they want to do with my code.

as long as it's complaint to AGPL v3 :D

(doubt ai companies care about that)

We want to give developers agency over their source code by letting them decide whether or not it should be used to develop and evaluate machine learning models.

fuck them seriously; if you want to do that then don't steal the repositories in the first place.

A decade ago, if someone asked someone working on an Open Source project if they'd be ok with an AI reading their code and learning from it, they'd most likely say "yeah that sounds really cool!"

Somehow the tech-bros have fucked up AI so much that something that should be really cool seems creepy, lame, and nefarious all at once.