this post was submitted on 19 Mar 2026

175 points (98.3% liked)

Technology

82830 readers

5197 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related news or articles.
Be excellent to each other!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
Check for duplicates before posting, duplicates may be removed
Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago

MODERATORS

L3s@lemmy.world

enu@lemmy.world

technopagan@lemmy.world

L4s@lemmy.world

L3s@hackingne.ws

175

AI still doesn't work very well, businesses are faking it, and a reckoning is coming (www.theregister.com)

submitted 15 hours ago by FoxtrotDeltaTango@sh.itjust.works to c/technology@lemmy.world

44 comments fedilink hide all child comments

cross-posted from: https://lemmy.ca/post/61948688

Excerpt:

"Even within the coding, it's not working well," said Smiley. "I'll give you an example. Code can look right and pass the unit tests and still be wrong. The way you measure that is typically in benchmark tests. So a lot of these companies haven't engaged in a proper feedback loop to see what the impact of AI coding is on the outcomes they care about. Lines of code, number of [pull requests], these are liabilities. These are not measures of engineering excellence."

Measures of engineering excellence, said Smiley, include metrics like deployment frequency, lead time to production, change failure rate, mean time to restore, and incident severity. And we need a new set of metrics, he insists, to measure how AI affects engineering performance.

"We don't know what those are yet," he said.

One metric that might be helpful, he said, is measuring tokens burned to get to an approved pull request – a formally accepted change in software. That's the kind of thing that needs to be assessed to determine whether AI helps an organization's engineering practice.

To underscore the consequences of not having that kind of data, Smiley pointed to a recent attempt to rewrite SQLite in Rust using AI.

"It passed all the unit tests, the shape of the code looks right," he said. It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."

All the optimism about using AI for coding, Smiley argues, comes from measuring the wrong things.

"Coding works if you measure lines of code and pull requests," he said. "Coding does not work if you measure quality and team performance. There's no evidence to suggest that that's moving in a positive direction."

you are viewing a single comment's thread
view the rest of the comments

[–] thebestaquaman@lemmy.world 22 points 14 hours ago (2 children)

It’s 3.7x more lines of code that performs 2,000 times worse than the actual SQLite.

Pretty much my experience with LLM coding agents. They'll write a bunch of stuff, and come with all kinds of arguments about why what they're doing is in fact optimal and perfect. If you know what you're doing, you'll quickly find a bunch of over-complicating things and just plain pitfalls. I've never been able to understand the people that claim LLMs can build entire projects (the people that say stuff like "I never write my own code anymore"), since I've always found it to be pretty trash at anything beyond trivial tasks.

Of course, it makes sense that it'll elaborate endlessly about how perfect its solution is, because it's a glorified auto-complete, and there's plenty of training data with people explaining why "solution X is better".

[–] Dojan@pawb.social 16 points 14 hours ago (2 children)

I saw a vibe coded PR the other day. So much redundant code, lots of comments making assumptions and questions. It’s a mess.

Glad it didn’t land in my lap but the person who is now responsible for steering that up is already quite busy and wasting their time with this feels shit.

[–] thebestaquaman@lemmy.world 8 points 12 hours ago

One of the worst things about this is that the person vibe coding just ends up shitting on the reviewers time. Like... you couldn't even bother to write a real PR, and now you want me to spend time filtering your shit? Fuck off.

[+] org@lemmy.org -8 points 14 hours ago (1 children)

To many people don’t know how to prompt AI, and review.

[–] Dojan@pawb.social 8 points 14 hours ago (1 children)

Too many people are willingly paying anti-democratic billionaires to outsource their thinking and agency.

[–] org@lemmy.org -5 points 14 hours ago (1 children)

Too many people know their job is only going to last 6 months before the next round of layoffs, and that talent and hard work has never been the way to keep a job in the tech industry… so why try?

[–] Dojan@pawb.social 6 points 14 hours ago (1 children)

Not really a valid excuse in this case as we aren’t really experiencing layoffs here. Au contraire, our company is hiring. I’m not in the U.S.

Still think that letting language models controlled by billionaire paedophiles and wannabe dictators is a poor idea, regardless of how fed up one is with one’s job.

[–] org@lemmy.org -4 points 13 hours ago (2 children)

Where is “here?”

And, if you want to bring pedophiles into it, most of what you touch on a daily basis involved a billionaire pedophile at some point. You just sound lazy at this point.

[–] EncryptKeeper@lemmy.world 6 points 12 hours ago* (last edited 12 hours ago) (1 children)

Did you just call them lazy after making the argument that building talent and hard work is not worth doing in the modern tech landscape?

[–] org@lemmy.org -4 points 12 hours ago

The argument is lazy to blanket everything in “pedophile” instead of actually talking about the issue.

[–] Dojan@pawb.social 1 points 12 hours ago (1 children)

Oh absolutely, and we can do our best to swear off of that but thanks to them worming their way in like a cancer in every part of society, shaping it to benefit them, that's just the nature of taking part in society. The ones in power have always, and will always continue to exploit us for as long as we let them.

All the more reason to not outsource our thinking to their machines. Governments are already doing it, getting caught red-handed acting on reports that never existed. Why rely on that when the option not to is so readily available?

[–] org@lemmy.org -3 points 12 hours ago (1 children)

Ehhh… this sounds more like blanket ai-hate and less about you actually caring. You’re already in their cloud. I doubt you run bare metal. You probably use GitHub. Etc. caring on one hand and not on the other means nothing.

I’ll continue farming out bullshit tasks to AI while I play with my cat and prepare for the next round of layoffs, rather than giving my soul to a company who doesn’t actually care about me.

[–] Dojan@pawb.social 1 points 11 hours ago

I actually really love machine learning. I trained my own language models back in the late 201Xs, and at my previous company I worked on image classification for a smaller photography platform they were developing. I'm not an ML expert, but it's easy to see the bullshit that the "AI" companies are selling as bullshit when you know the foundations of the tech. A little like how you don't need to be a surgeon to call bullshit on someone saying that they performed open heart surgery and brain surgery on themselves simultaneously.

You’re already in their cloud. Yes, unfortunately. There was a time when I was a lot more naïve and way less critical. People change. This is what "radicalised" me. I know that a story like that is just daily life in the U.S., but despite my cynicism I thought things were better here.

You probably use GitHub. I don't. I self-host everything that I can, and make careful choices with what I choose not to. I've left what I had on GitHub on there, and I'll probably use it as a mirror whenever I release a FOSS project, because I'm OK with Microsoft paying the hosting costs for me, if they're going to try and scrape my shit anyway. You know, just like GNOME is doing.

caring on one hand and not on the other means nothing.

I don't really agree. This is the same idiotic take as "if you hate capitalism so much, why are you partaking in it?" You're also speaking directly through your sphincter as you've no idea what my life is like, what choices I've made, and so forth; you don't know me.

Would I prefer to never have to engage with a payment processor or a bank again? Absolutely. That however, is sadly impossible in my society. LLMs aren't integral to society yet, and I'd see them continue that way.

I’ll continue farming out bullshit tasks to AI while I play with my cat and prepare for the next round of layoffs, rather than giving my soul to a company who doesn’t actually care about me.

Not a fan of the LLM part, but I love the overall sentiment. Fuck the corporations, your time and energy is better spent on the people you love. I hope things work out for you.

~Animals are people too. Lots of love to your cat.~

~I love cats.~

[–] thisbenzingring@lemmy.today 5 points 14 hours ago (1 children)

I tried using an LLM for making an 3d object in openscad, an open source CAD app for making 3d printable objects

its basic and uses an open source language. The LLM should have infinate examples and access

but after 4 tries I gave up and just did it myself, sure the crap the LLM gave me helped form a general setup but I had to spend 2x as much time fixing the code then it did writing it from scratch

I haven't tried using LLM for anything else, that failure told me everything I needed to know about its ability to do basic shit

[–] grue@lemmy.world 0 points 14 hours ago

I have never heard of any generative AI system capable of doing anything useful with 3D models. If you ever find one, PM me to let me know!