"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
link to archived Reddit thread; original post removed/deleted
I've said it time and time again: AIs aren't trained to produce correct answers, but seemingly correct answers. That's an important distinction and exactly what makes AIs so dangerous to use. You will typically ask the AI about something you yourself are not an expert on, so you can't easily verify the answer. But it seems plausible so you assume it to be correct.
Thankfully, AI is bad at maths for exactly this reason. You don't have to be an expert on a very specific topic to be able to verify a proof and - spoiler alert - most of the proofs ChatGPT 5 has given me are plain incorrect, despite OpenSlop's claims that it is vastly superior to previous models.
My own advise for people starting to use AI is to use it for things you know very well. Using it for things you do not know well, will always be problematic.
I suspect this will happen all over with in a few years, AI was good enough at first, but over time reality and the AI started drifting apart
AI is literally trained to get the right answer but not actually perform the steps to get to the answer. It's like those people that trained dogs to carry explosives and run under tanks, they thought they were doing great until the first battle they used them in they realized that the dogs would run under their own tanks instead of the enemy ones, because that's what they were trained with.
Holy shit, that's what they get for being so evil that they trained dogs as suicide bombers.
And then, the very same CEOs that demanded the use of AI in decision making will be the ones that blame it for bad decisions.
while also blaming employees
Of course, it is the employees who used it. /s
They haven't drifted apart, they were never close in the first place. People have been increasingly confident in the models because they've increasingly sounded more convincing, but the tenuous connection to reality has been consistently off.
I raised this as a concern at the corporate role I work in when an AI tool that was being distributed and encouraged for usage showed two hallucinated data points that were cited in a large group setting. I happened to know my area well, the data was not just marginally wrong but way off, and I was able to quickly check the figures. I corrected it in the room after verifying on my laptop and the reaction in the room was sort of a harmless whoops. The rest of the presentation continued without a seeming acknowledgement that the rest of the figures should be checked.
When I approached the head of the team that constructed the tool after the meeting and shared the inaccuracies and my concerns, he told me that he'd rather have more data fluency through the ease of the tool and that inaccuracies were acceptable because of the convenience and widespread usage.
I suspect stories like this are happening across my industry. Meanwhile, the company put out a press release about our AI efforts (literally using Gemini's Gem tool and custom ChatGPTs seeded with Google Drive) as something investors should be very excited about.
When I approached the head of the team that constructed the tool after the meeting and shared the inaccuracies and my concerns, he told me that he’d rather have more data fluency through the ease of the tool and that inaccuracies were acceptable because of the convenience and widespread usage.
"I prefer more data that's completely made up over less data that is actually accurate."
This tells you everything you need to know about your company's marketing and data analysis department and the whole corporate leadership.
Potemkin leadership.
Honestly this is not a new problem and is a further expression of the larger problem.
"Leadership" becomes removed from the day to day operations that run the organization and by nature the "cream" that rises tend to be sycophantic in nature. Our internal biases at work so it's no fault of the individual.
Humanity is their own worst enemy lol
It is not a new problem and that has been the case for a long time. But it's a good visualization of it.
Everyone in a company has their own goals, from the lowly actual worker who just wants to pay the bills and spend as little effort on it as possible, to departments which want to justify their useless existence, to leadership who mainly wants to look good towards the investors to get a nice bonus.
That some companies end up actually making products that ship and that people want to use is more of an unintended side effect than the intended purpose of anyone's work.
That makes no sense. The inaccuracies are even less acceptable with widespread use!
I somehow hope this is made up, because doing this without checking and finding the obvious errors is insane.
This is probably real, as it isn't the first time it happened: https://www.theguardian.com/technology/2025/jun/06/high-court-tells-uk-lawyers-to-urgently-stop-misuse-of-ai-in-legal-work
As someone who has to deal with LLMs/AI daily in my work in order to fix the messes they create, this tracks.
AI's sole purpose is to provide you a positive solution. That's it. Now that positive solution doesn't even need to be accurate or even exist. It's built to provide a positive "right" solution without taking the steps to get to that "right" solution thus the majority of the time that solution is going to be a hallucination.
you see it all the time. you can ask it something tech related and in order to get to that positive right solution it'll hallucinate libraries that don't exist, or programs that don't even do what it claims they do. Because logically to the LLM this is the positive right solution WITHOUT utilizing any steps to confirm that this solution even exists.
So in the case of OPs post I can see it happening. They told the LLM they wanted analytics for 3 months and rather than take the steps to get to an accurate solution it ignored said steps and decided to provide positive solution.
Don't use AI/LLMs for your day to day problem solving. you're wasting your time. OpenAI, Anthropic, Google, etc have all programmed these things to provide you with "positive" solutions so you'll keep using them. they just hope you're not savvy enough to call out their LLM's when they're clearly and frequently wrong.
Probably the skepticism is around someone actually trusting the LLM this hard rather than the LLM doing it this badly. To that I will add that based on my experience with LLM enthusiasts, I believe that too.
I have talked to multiple people who recognize the hallucination problem, but think they have solved it because they are good "prompt engineers". They always include a sentence like "Do not hallucinate" and thinks that works.
The gaslighting from the LLM companies is really bad.
Use of AI in companies would not save any time if you were checking each result.
It has happened in the open, so I don't see why it wouldn't happen even more behind closed doors:
Deloitte will provide a partial refund to the federal government over a $440,000 report that contained several errors, after admitting it used generative artificial intelligence to help produce it..
I work in a regulated sector and our higher ups are pushing AI so much. And there response to AI hallucinations is to just put a banner on all internal AI tools to cross verify and have some quarterly stupid "trainings" but almost everyone I know never checks and verifies the output. And I know of atleast 2 instances where because AI hallucinated some numbers we sent out extra money to a third party.
My workplace (finance company) bought out an investments company for a steal because they were having legal troubles, managed to pin it on a few individuals, then fired the individuals under scrutiny.
Our leadership thought the income and amount of assets they controlled was worth the risk.
This new group has been the biggest pain in the ass. Complete refusal to actually fold into the company culture, standards, even IT coverage. Kept trying to sidestep even basic stuff like returning old laptops after upgrades.
When I was still tech support, I had two particularly fun interactions with them. One was when it was discovered that one of their top earners got fired for shady shit, then they discovered a month later that he had set his mailbox to autoreply to every email pointing his former clients to his personal email. Then, they hired back this guy and he lasted a whole day before they caught him trying to steal as much private company info as he could grab. The other incident was when I got a call from this poor intern they hired, then dumped the responsibility for this awful home grown mess of Microsoft Access, Excel, and Word docs all linked over ODBC on this kid. Our side of IT refused to support it and kept asking them to meet with project management and our internal developers to get it brought up into this century. They refused to let us help them.
In the back half of last year, our circus of an Infosec Department finally locked down access to unapproved LLMs and AI tools. Officially we had been restricted to one specific one by written policy, signed by all employees, for over a year but it took someone getting caught by their coworker putting private info into a free public chatbot for them to enforce it.
Guess what sub-company is hundreds of thousands of dollars into a shadow IT project that has went through literally none of the proper channels to start using an explicitly disallowed LLM to process private customer data?
As an unemployed data analyst / econometrician:
lol, rofl, perhaps even... lmao.
Nah though, its really fine, my quality of life is enormously superior barely surviving off of SSDI and not having to explain data analytics to thumb sucking morons (VPs, 90% of other team leads), and either fix or cover all their mistakes.
Yeah, sure, just have the AI do it, go nuts.
I am enjoying my unexpected early retirement.
Joke's on you, we make our decisions without asking AI for analytics. Because we don't ask for analytics at all
I feel like no analytics is probably better than decisions based on made-up analytics.
I don't need AI to fabricate data. I can be stupid on my own, thank you.
When you delegate, to a person, a tool or a process, you check the result. You make sure that the delegated tasks get done and correctly and that the results are what is expected.
Finding that it is not the case after months by luck shows incompetence. Look for the incompetent.
Before anything else: whether the specific story in the linked post is literally true doesn’t actually matter. The following observation about AI holds either way. If this example were wrong, ten others just like it would still make the same point.
What keeps jumping out at me in these AI threads is how consistently the conversation skips over the real constraint.
We keep hearing that AI will “increase productivity” or “accelerate thinking.” But in most large organizations, thinking is not the scarce resource. Permission to think is. Demand for thought is. The bottleneck was never how fast someone could draft an email or summarize a document. It was whether anyone actually wanted a careful answer in the first place.
A lot of companies mistook faster output for more value. They ran a pilot, saw emails go out quicker, reports get longer, slide decks look more polished, and assumed that meant something important had been solved. But scaling speed only helps if the organization needs more thinking. Most don’t. They already operate at the minimum level of reflection they’re willing to tolerate.
So what AI mostly does in practice is amplify performative cognition. It makes things look smarter without requiring anyone to be smarter. You get confident prose, plausible explanations, and lots of words where a short “yes,” “no,” or “we don’t know yet” would have been more honest and cheaper.
That’s why so many deployments feel disappointing once the novelty wears off. The technology didn’t fail. The assumption did. If an institution doesn’t value judgment, uncertainty, or dissent, no amount of machine assistance will conjure those qualities into existence. You can’t automate curiosity into a system that actively suppresses it.
Which leaves us with a technology in search of a problem that isn’t already constrained elsewhere. It’s very good at accelerating surfaces. It’s much less effective at deepening decisions, because depth was never in demand.
If you’re interested, I write more about this here: https://tover153.substack.com/
Not selling anything. Just thinking out loud, slowly, while that’s still allowed.
I'm a data analyst and primary authority on the data model of a particular source system. Most questions for figures from that system that can't be answered directly and easily in the frontend end up with me.
I had a manager show me how some new LLM they were developing (which I had contributed some information about the model to) could quickly answer some questions that usually I have to answer manually, as part of a pitch to make me switch to his department so I can apply my expertise for improving this fancy AI instead of answering questions manually.
He entered a prompt, got a figure that I knew wasn't correct and I queried my data model for the same info, with a significantly different answer. Given how much said manager leaned on my expertise in the first place, he couldn't very well challenge my results and got all sheepish about how the AI still in development and all.
I don't know how that model arrived at that figure. I don't know if it generated and ran a query against the data I'd provided. I don't know if it just invented the number. I don't know how the devs would figure out the error and how to fix it. But I do know how to explain my own queries, how to investigate errors and (usually) how to find a solution.
Anyone who relies on a random text generator - no matter how complex that generation method to make it sound human - to generate facts is dangerously inept.
To everyone I've talked to about AI, I've suggested a test. Take a subject that they know they are an expert at. Then ask AI questions that they already know the answers to. See what percentage AI gets right, if any. Often they find that plausible sounding answers are produced however, if you know the subject, you know that it isn't quite fact that is produced. A recovery from an injury might be listed as 3 weeks when it is average 6-8 or similar. Someone who did not already know the correct information, could be damaged by the "guessed" response of AI. AI can have uses but it needs to be heavily scrutinized before passing on anything it generates. If you are good at something, that usually means you have to waste time in order to use AI.
This would suggest the leadership positions aren't required for the function of the business.
This has always been the case, in every industry.
Jesus Christ, you have to have a human validate the data.
Exactly, this is like letting excel auto-fill finish the spreadsheet and going "looks about right"
And that's a good analogy, as people have posted screenshots of Copilot getting basic addition wrong in Excel.
Whoever implemented this agent without proper oversight needs to be fired.
It doesn't matter. Management wants this and will not stop until they run against a wall at full speed. 🤷
My broseph in Christ, what did you think a LLM was?
Bro, just give us a few trillion dollars, bro. I swear bro. It'll be AGI this time next year, bro. We're so close, bro. I just need need some money, bro. Some money and some god-damned faith, bro.
Dumbasses. Mmm, that's good schadenfreude.
But don't worry, when it comes to life or death issues, AI is the best way to help
Haha, "chat, how do I stop the patients nose from bleeding"
"Cut his leg off."
"Well, you're the medicAI. Nurse, fetch the bonesaw"
My workplace, the senior management, is going all in on Copilot. So much so that at the end of last year to told us to use Copilot for year end reviews! Even provided a prompt to use, told us to link it to Outlook (not sure why, since our email retention isn't very long)... but whatever.
I tried it, out of curiosity because I had no faith. It started printing out stats for things that never happened. It provided a 35% increase here, a 20% decress there, blah blah blah. It didn't actually highlight anything I do or did. And I'm banking that a human will partially read my review, not just use AI.
If someone read it, I'm good. If AI reads it, I do wonder if I screwed myself. Since senior mgmt is just offloading to AI...
Ah yes, what a surprise. The random word generator gave you random numbers that aren't actually real.
Our AI that monitors customer interactions sometimes makes up shit that didn't happen during the call. Any agent smart enough could probably fool it into giving the wrong summary with the right key words. I only caught onto it when I started reading the logs carefully, but I don't know if management cares so long as the business client is happy.
Apparently that reddit post itself was generated with AI. Using AI to bash AI is an interesting flex.