diz

So, I was thinking about how modern LLMs got better at coding in the last 1 year, despite not having much good new training data.

The way it was done is largely via reinforcement learning with verifiable reward (and a little bit with human feedback).

Basically, you use an LLM to make slop, random token choice and everything, monkeys on typewriters style, and then if slop passes a test, you use said slop as the training data. You use the compiler, unit and integration tests, any static analysis tools, etc etc as verification.

For natural language, attempts to do anything like this work miserably, the whole slop centipede, model collapse, all that. But for coding this turned to be unexpectedly effective.

So that raises the question: can an LLM coding assistant be developed like a normal corporate project, with no YOLO exemptions from software legal compliance, without any special treatment like it's a favorite child that grows up to be a techbro? Like normal fucking software?

I think, the answer is probably a yes on the purely technical level but still a hard no commercially.

On the technical level, an LLM could be pre-trained on compatibly licensed open source projects, RLVR'd in a license-compliant way.

The end result isn't going to be good for plagiarism, though, and won't use plagiarism to cover every gap in capabilities. I think it's pretty obvious all the folks who are trying to clone Adobe or whatever would reject such a product.

Meanwhile the one "strength" of LLMs is this intuitive argument that if you have a good enough predictor of what your workers do, then you don't need workers. That is the whole reason the "AI companies" are rolling in investor money and are able to operate at such large loss.

The other thing is that having done copyright infringement, OpenAI/Anthropic/etc etc have to portray it as necessary, and alternative as impossible.

Anyhow basically my thought on this is that I'll continue not using any LLM coding tools in my personal projects until some acceptable tooling appears and proves itself actually useful.

edit: For a concrete example if someone makes a tool that can take the compiled assembly, run it through non-LLM translation to some much abridged Lean and prove that nothing depends on exact timing of threads (no race conditions) or identify race conditions, or complain it could do neither, would I use it? If the carbon footprint is not insanely greater than what I would do instead (which is either a: run stress tests overnight, or b: ship buggy software that users have to restart, which also has a large energy footprint), sure.

As I was using LLMs at work (I never use LLMs for personal projects), a question occurred to me:

What in the fuck are these new data centers supposed to actually do?

The enterprise users are all in on having LLMs write 300% of the code. And yet, both OpenAI and Anthropic are still giving these "$200 gets you $10 000 in API costs" subscriptions to every victim of AI psychosis.

Why do we have RAM, hard disk, SSD, GPU and CPU shortages and no "token shortage"?

I mean, we all know the answer, these are the offerings for the robot god, but like how in the fuck are they still obtaining loans for this shit?

Took a peek at what our good friends the rationalists are talking about. That's gonna teach me not to gaze into the abyss.

Apparently some AI related bigshot CEO is telling Stuart Russell that our best hope for survival is an "AI Chernobyl" which would royally piss everyone off even more.

Our "leaders" are quaking in their boots and hoping a bunch of us would die so we the people take the reins and lead them somehow out of a peril of their own construction. That's whole another level of completely fucked up.

You all know I'm the last person to expect this shit to get 3 vegetables across the river without going back, and yet the latest evolution of their bullshit is making me feel rather doomer-y.

Reading between the lines, Chornobyl = chain reaction = forkbomb = agents hacking their own company to launch more agents with the same prompt. Do I think this is possible? Sure as shit I think its possible. their cybersecurity is terrible. And LLMs did turn out great at finding exploits: humans screw up in stereotypical ways and there's a gazillion "fixed blah blah" git commits to train on.

Okay, so we got a tremendous cluster fuck. Ideally all that happens is another "human coding appreciation day", whatever.

Except as can be told from their recent hacking leaks, what happens when a clusterfuck like this runs, with the context full of hacking, it would keep trying to hack whatever, until stopped. Hopefully, "whatever" is other AI companies, but it's not like they can't train or fine tune to diversify the whatever.

Other bit that is very disturbing to me is that they are talking about a billion deaths scenario, instead of talking about outright extinction. Why the sudden modesty?

My cynical take is at the core their concerns are still insincere, but this time they actually think something serious might happen, not just having fun with fantasies of infinite importance. And so they are inventing new scenarios that would make that serious thing seem like we got off lightly, or perhaps turn it into a regulatory capture opportunity.

Which is just an escalation, or perhaps de-escalation, of their old game of talking over any realistic concerns with "existential risk".

A quantum sweatshop.

There's a very long history of extremely effective labor saving tools in software.

Writing in C rather than Assembly, especially for more than 1 platform.

Standard libraries. Unix itself. More recently, developing games in Unity or Unreal instead of rolling your own engine.

And what happened when any of these tools come on the scene is that there is a mad gold rush to develop products that weren't feasible before. Not layoffs, not "we don't need to hire junior developers any more".

Rank and file vibe coders seem to perceive Claude Code (for some reason, mostly just Claude Code) as something akin to the advantage of using C rather than Assembly. They are legit excited to code new things they couldn't code before.

Boiling the rivers to give them an occasional morale boost with "You are absolutely right!" is completely fucked up and I dread the day I'll have to deal with AI-contaminated codebases, but apart from that, they have something positive going for them, at least in this brief moment. They seem to be sincerely enthusiastic. I almost don't want to shit on their parade.

The AI enthusiast bigwigs on the other hand, are firing people, closing projects, talking about not hiring juniors any more, and got the media to report on it as AI layoffs. They just gleefully go on about how being 30% more productive means they can fire a bunch of people.

The standard answer is that they hate having employees. But they always hated having employees. And there were always labor saving technologies.

So I have a thesis here, or a synthesis perhaps.

The bigwigs who tout AI (while acknowledging that it needs humans for now) don't see AI as ultimately useful, in the way in which C compiler was useful. Even if its useful in some context, they still don't. They don't believe it can be useful. They see it as more powerfully useless. Each new version is meant to be a bit more like AM or (clearly AM-inspired, but more familiar) GLaDOS, that will get rid of all the employees once and for all.

Sounds like meta’s judge will have to invent a grand unified theory of fair use to excuse this.

I kept saying about various lawsuits that the important thing is discovery. Nobody knew all the idiotic shit these folks were doing, so nobody could sue them properly.

They train on sneer-problems now:

Here’s the “ferry‑shuttle” strategy, exactly analogous to the classic two‑ferryman/many‑boats puzzle, but with planes and pilots

And lo and behold, singularity - it can solve variants that no human can solve:

https://chatgpt.com/share/68813f81-1e6c-8004-ab95-5bafc531a969

Two ferrymen and three boats are on the left bank of a river. Each boat holds exactly one man. How can they get both men and all three boats to the right bank?

I think this summarizes in one conversation what is so fucking irritating about this thing: I am supposed to believe that it wrote that code.

No siree, no RAG, no trickery with training a model to transform the code while maintaining identical expression graph, it just goes from word-salading all over the place on a natural language task, to outputting 100 lines of coherent code.

Although that does suggest a new dunk on computer touchers, of the AI enthusiast kind, you can point at that and say that coding clearly does not require any logical reasoning.

(Also, as usual with AI it is not always that good. sometimes it fucks up the code, too).

I love to show that kind of shit to AI boosters. (In case you're wondering, the numbers were chosen randomly and the answer is incorrect).

They go waaa waaa its not a calculator, and then I can point out that it got the leading 6 digits and the last digit correct, which is a lot better than it did on the "softer" parts of the test.

I couldn't stop fucking laughing. I'm wheezing. It's unhealthy.

They have this thing acting like that for the whole day... and then more than a day later claim it was hacked.

Tried my duck river crossing thing a few times recently, it usually solves it now, albeit with a bias to make unnecessary trips half of the time.

Of course, anything new fails:

There's 2 people and 1 boat on the left side of the river, and 3 boats on the right side of the river. Each boat can accommodate up to 6 people. How do they get all the boats to the left side of the river?

Did they seriously change something just to deal with my duck puzzle? How odd.

It's Google so it is not out of the question that they might do some analysis on the share links and referring pages, or even use their search engine to find discussions of a problem they're asked. I need to test that theory and simultaneously feed some garbage to their plagiarism machine...

Sample of the new botshit:

L->R: 2P take B_L. L{}, R{2P, 4B}. R->L: P1 takes B_R1. L{P1, B_R1}, R{P2, 3B}. R->L: P2 takes B_R2. L{2P, B_R1, B_R2}, R{2B}. L->R: P1 takes B_R1 back. L{P2, B_R2}, R{P1, 3B}. R->L: P1 takes B_R3. L{P1, P2, B_R2, B_R3}, R{2B}. L->R: P2 takes B_R2 back. L{P1, B_R3}, R{P2, 3B}.

And again and again, like a buggy attempt at brute forcing the problem.