MangoCats

There are no guarantees in life, especially when dealing with people you've never met on the other side of the planet through the internet.

However, when an LLM finds a real bug or vulnerability, somewhat akin to an NP-hard problem, you can much more easily verify that the problem is real once it has been identified - you can do the verification with or without LLM assistance.

will not fall to some prompt injection that can be within the code?

that is a concern, one which has been identified and is in a state of continual improvement to make less of a concern in LLM agents / harnesses. It's also not without human based precedent, with misleading comments and even obfuscated code structure leading humans to believe that code does something different than what it looks like to a casual or fast/shallow reader. The ability of LLM agents to more quickly run many actual tests of the code than a human tester is a huge advantage in determining what code actually does, and identifying rarely triggered pathways that may have concerning functionality.

I upped from the $20/month subscription to $200/month for a few months, it was pretty hard to run out of tokens on the $200/month plan - and have any life away from the keyboard at the same time.

Harder for the people who want backdoors or critical security bugs approved, I'd say.

That promise of open source is getting a little boost here: many eyes on the code, insecure patterns revealed for lower effort in less time - able to be caught before wide release because the (robot) reviewers have the capacity to run the code through many more test scenarios before delivering a verdict.

I don't view capitalism as the primary cause, but more as the excuse. We don't have free unfettered capitalism, there are price controls, labor regulations, every kind of distortion of capitalism: when it serves the agendas at play. When the agenda doesn't care to protect a certain class, like diabetics with not much money, then the agenda lets capitalism run wild.

So, sure, blame capitalism - it's what happens when you let people decide how much profit they want in combination with IP protections and monopoly markets. That way you don't have to point the finger at a "communist" central planning committee that arbitrarily decided the price for something. In the end, it's still people making decisions. As the song goes: when you choose not to decide, you still have made a choice.

The XZ Utils backdoor is a very recent example of a contributor trying to abuse a backdoor he put there himself, after 2 years of posing as a helpful person.

A game that has been played for decades. A game that LLM review of merge requests should be making very much harder to play in the future.

The thing is: the review process now also has AI, and those AI assisted reviews are much more stringent than the traditional human only reviews. Neither are perfect, ideally both would give the submission all the attention it deserves - in real life that rarely happens, though at the moment the LLMs are fast and cheap enough that you really should let them review anything you are considering approving for merge - not necessarily take everything they output at face value, but I find that they find real problems that real humans often miss.

it’s all 's fault who is a fucking imbecile

And, as always, he's not wrong.

when the day comes, and it will, when something is submitted and approved and breaks shit

That day has been coming, over 300 times a year for the past 30 years. People submit things that break shit all-the-time, the system is setup to deal with it, that's the actual genius of FOSS development procedures, they're built to handle it.

Now, the firehose isn't just opened, the hydrant has been knocked off entirely, and there will be some growing pains to deal with the volume, but the basic procedures are well developed and I find that applying them to LLM outputs (before passing those outputs anywhere off my own desk) goes a very long way toward refining slop into genuinely robust solutions.

bogus bug reports “discovered by AI” that could not be replicated in any way, shape or form

And we NEED to be more serious about use of secure verifiable identities for people so we can properly filter inputs from internet contributions. Anonymous sources should be de-prioritized to just above the known consistently bad actors. That's nothing about AI or LLMs, that's just the world continuing to be far too trusting of basically anonymous contributions.

If you want to submit a bunch of AI slop under your secure ID, be prepared for that ID to be de-prioritized beneath anonymous contributions.

If you want submissions from your secure ID to be taken seriously, at least task your LLM with building a proof-of-concept replication of the bugs in the reports your other LLM session creates before submitting it for humans to spend their time evaluating.

We have a human test department overseas that gets a little eager to pump up their metrics once in a while, we haven't resorted to penalizing them (yet) but when they send an "observation" with no way to reproduce it, no log files to back it up, we tend to ignore that observation until the review meeting and then in the review meeting ask "have you seen this again? do you have any way to reproduce it? do you have screen shots or log files?" and when all those things are missing, the observation is closed with a "cannot reproduce" tag, waiting for another tester to see it again and produce more evidence.

the article also seems overly confident

Anybody writing articles about AI, or evaluating articles about AI, needs to get very specific: what month are they talking about, because if your reference point for "overly confident" is based on where things were 8-12 months ago, that's very different from where things are today, and many times experts in the field project progress to what they expect for the future - that is of course unreliable speculation, but so far most of the reasonable optimism from the last year or two for progress in LLM software authorship and maintenance proficiency has been borne out in reality. I'm not talking about the "software is solved" numbskulls, but the progress in terms of being a valuable tool set the way C compilers were valuable tools to former Assembly programmers - yeah... it's gone there.

I also recall trying to use Borland Turbo C++ in 1991 and abandoning it because it was too buggy, wrote the project du jour in C instead, saved the C++ rewrite for 1996.

just because AI can find a working pattern doesn’t make it good

It's a wee bit more sophisticated than that, and will tend to find 2 + 2 even when 16/4 - 8/4 gives the same result. And if you suspect it's settled on an over-complicated solution you can add the prompt: "Search for simpler architectures which meet all requirements with less complexity." It won't always find the simplest possible, but generally speaking - I find LLM agents to be a bit better than my human colleagues in terms of finding good simple solutions and also finding problems within implementations.

None of us are perfect, and LLMs are trained on our writings. LLM agents within coding frameworks have progressed easily +10 years of human experience in their perfomance levels in the last 12 months. It's still an blueberries to tangerines comparison, a lot of what makes the LLM agents "good" is their speed and thereby ability to experiment with alternatives in a reasonable time.

Now we don’t know who actually wrote the code

Did you ever? Really?

On the internet nobody knows you're a dog.