TechTakes

I knew the Anthropic blog post was bullshit but every single time the reality is 10x worse that I anticipated.

@sailor_sega_saturn @dovel the nice thing about rogue chatbots is that there's no difference between that and a non-rogue chatbot

The first issue filed is called “Hello world does not compile” so you can tell it’s off to a good start.

The comments are a hoot, at least.

I wonder what actual experts in compilers think of this.

Anthropic doesn't pay me and I'm not going to look over their pile of garbage for free, but just looking at the structure and READMEs it looks like a reasonable submission for an advanced student in a compiler's course: lowering to IR, SSA representation, dominators, phi elimination, some passes like strength reduction. The register allocator is very bad though, I'd expect at least something based on colouring.

The READMEs are also really annoying to read. They are overlong and they don't really explain what is going on in the module. There's no high-level overview of the architecture of the compiler. A lot of it is just redundant. Like, what is this:

Ye dude, of course it doesn't depend on the IR, because this is before IR is constructed. Are you just pretending to know how a compiler works? Wait, right, you are, you're a bot. The last sentence is also hilarious, my brother in christ, what, why is this in the README.

Now this evaluation only makes sense if the compiler actually works - which it doesn't. Looking at the filed issues there are glaring disqualifying problems (#177, #172, #171, #167, etc. etc. etc.). Like, those are not "oops, forgot something", those are "the code responsible for this is broken". Some of them look truly baffling, like how do you manage to get so many issues of the type "silently does something unexpected on error" when the code is IN RUST, which is explicitly designed to make those errors as hard as possible? Like I'm sorry, but the ones below? These are just "you did not even attempt to fulfill the assignment".

It's also not tested, it has no integration tests (even though the README says it does), which is plain unacceptable. And the unit tests that are there fail so lol, lmao.

It's worse than existing industry compilers and it doesn't offer anything interesting in terms of the implementation. If you're introducing your own IR and passes you have to have a good enough reason to not just target LLVM. Cranelift is... not great, but they at least have interesting design choices and offer quick unoptimized compilation. This? The only reason you'd write this is you were indeed a student learning compilers, in which case it'd be a very good experience. You'd probably learn why testing is important for the rest of your life at least.

I wonder why this blog post was brazen enough to talk about these problems. Perhaps by throwing in a little humility, they can make the hype pill that much easier to swallow.

I feel this is an artefact of the near complete collapse of mainstream journalism, combined with modern tech business practises that are about securing investment and cashing out, and every other concern is secondary or even entirely absent. It’s all just selling vibes.

People only ever report the hype, the investors see everyone else following the hype and panic that they might be left out and bury you in cash. When it all turns sour and people ask pointed questions about the exact nature of the magic beans you were promising to grow, you can just point at the blog post that no-one read (or at least, only poor people read, and they’re barely people if you think about it) and point out that you never hid anything.

I only sampled some of the docs and interesting-sounding modules. I did not carefully read anything.

First, the user-facing structure. The compiler is far too configurable; it has lots of options that surely haven't been tested in combination. The idea of a pipeline is enticing but it's not actually user-programmable. File headers are guessed using a combination of magic numbers and file extensions. The dog is wagged in the design decisions, which might be fair; anybody writing a new C compiler has to contend with old C code.

Next, I cannot state enough how generated the internals are. Every hunk of code tastes bland; even when it does things correctly and in a way which resembles a healthy style, the intent seems to be lacking. At best, I might say that the intent is cargo-culted from existing code without a deeper theory; more on that in a moment. Consider these two hunks. The first is generated code from my fork of META II:

while i < len(self.s) and self.clsWhitespace(ord(self.s[i])): i += 1

And the second is generated code from their C compiler:

while self.pos < self.input.len() && self.input[self.pos].is_ascii_whitespace() {
    self.pos += 1;
}

In general, the lexer looks generated, but in all seriousness, lexers might be too simple to fuck up relative to our collective understanding of what they do. There's also a lot of code which is block-copied from one place to another within a single file, in lists of options or lists of identifiers or lists of operators, and Transformers are known to be good at that sort of copying.

The backend's layering is really bad. There's too much optimization during lowering and assembly. Additionally, there's not enough optimization in the high-level IR. The result is enormous amounts of spaghetti. There's a standard algorithm for new backends, NOLTIS, which is based on building mosaics from a collection of low-level tiles; there's no indication that the assembler uses it.

The biggest issue is that the codebase is big. The second-biggest issue is that it doesn't have a Naur-style theory underlying it. A Naur theory is how humans conceptualize the codebase. We care about not only what it does but why it does. The docs are reasonably-accurate descriptions of what's in each Rust module, as if they were documents to summarize, but struggle to show why certain algorithms were chosen.

Choice sneer, credit to the late Jessica Walter for the intended reading: It's one topological sort, implemented here. What could it cost? Ten lines?

I do not believe that this demonstrates anything other than they kept making the AI brute force random shit until it happened to pass all the test cases.

That's the secret: any generative tool which adapts to feedback can do that. Previously, on Lobsters, I linked to a 2006/2007 paper which I've used for generating code; it directly uses a random number generator to make programs and also disassembles programs into gene-like snippets which can be recombined with a genetic algorithm. The LLM is a distraction and people only prefer it for the ELIZA Effect; they want that explanation and Naur-style theorizing.

I wonder if this is going to hold out long enough to get some obnoxious AI-first language created that is designed to have as obnoxiously picky of a compiler as it can in order to try and turn runtime errors that the model can't cope with into compile failures which it can silently retry until they're 'fixed'

This could be regarded as a neat fun hack, if it wasn’t built by appropriating the entire world of open source software

This shouldn't be left merely implied, the autoplag trained on GCC, clang, and every single poor undergrad who had to slap together a working C compiler for their compilers course and uploaded it to github, and "learnt" fuckall

did you have a point you wanted to make?