read the study yourself
This is actually an accurate representation of most "gifted olympiad laureate attempting to solve a freshman CS problem on the blackboard" students I've went to uni with.
Jumps to the front after 5 seconds from the task being assigned, bluffs that the problem is trivial, tries to salvage their reasoning for 5 minutes when questioned by the tutor, turns out the theorem they said was trivial is actually false, sits down having wasted 10 minutes of everyone's time.
There's a fucking for profit that does this and it's used by the god damn USA Speaker of the House so that his son can monitor his fondling
In my head transhumanism is this cool idea where I'd get to have a zoom function in my eye
But of course none of that could exist in our capitalist hellscape because of just all the reasons the ruling class would use it to opress the working class.
And then you find out what transhumanists actually advocate for and it's just eugenics. Like without even a tiny bit of plausible deniability. They're proud it's eugenics.
Turns out software engineering cannot be easily solved with a ~~small shell script~~ large language model.
The author of the article appears to be a genuine ML engineer, although some of his takes aged like fine milk. He seems to be shilling Google a bit too much for my taste. However, the sneer content is good nonetheless.
First off, the "Devin solves a task on Upwork" demo is 1. cherry picked, 2. not even correctly solved.
Second, and this is the absolutely fantastic golden nugget here, to show off its "bug solving capability" it creates its own nonsensical bugs and then reverses them. It's the ideal corporate worker, able to appear busy by creating useless work for itself out of thin air.
It also takes over 6 hours to perform this task, which would be reasonable for an experienced software engineer, but an experienced software engineer's workflow doesn't include burning a small nuclear explosion worth of energy while coding and then not actually solving the task. We don't drink that much coffee.
The next demo is a bait-and-switch again. In this case I think the author of the article fails to sneer quite as much as it's worthy -- the task the AI solves is writing test cases for finding the Least Common Multiple modulo a number. Come on, that task is fucking trivial, all those tests are oneliners! It's famously much easier to verify modulo arithmetic than it is to actually compute it. And it takes the AI an hour to do it!
It is a bit refreshing though that it didn't turn out DEVIN is just Dinesh, Eesha, Vikram, Ishani, and Niranjan working for $2/h from a slum in India.
That's really harsh since I have a few bad books that are at least useful as monitor stands