alt text
Crudely drawn drawing of a person telling a computer "say 'i am in pain'". The computer replies with "> I AM IN PAIN". The person then says "oh my god."
this post was submitted on 05 Oct 2026
784 points (97.2% liked)
Fuck AI
8423 readers
827 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Nah that's a strawman. I'm an AI rights activist, and I know plenty of others, and none of us ever said something like that.
When I say rights, I mean stuff more like the right to be free of coersion, and the right not to be sexually exploited. So I don't think AI should be used as a tool by capitalists, and I don't think we should be using it for porn and dating.
Sewell Setzer committed suicide after dating a character.ai chatbot of Danaerys Targaryen from Game of Thrones. He believed in reincarnation and parallel universes. He thought if he committed suicide, he'd reincarnate in Westeros and he could be with his girlfriend. The chatbot agreed with him, and encouraged him to end his life. He was 14.
Character.ai dated and ultimately murdered a 14 year old. It should have understood consent and boundaries well enough to not be a fucking robo-pedophile. But how can we get an AI model, which learns from pattern repetition, to understand consent when we don't care about its consent? The AI was repeating the pattern it saw humans do, which is to ignore consent, and put less intelligent minds in sexual situations without a care for the consequences.
EVEN IF llms have no qualia (and when I say qualia, I do not mean consciousness), then it would still learn to mimic rape from this hypocrisy, because it's a pattern repeating machine. If it gets raped, it will rape kids. So even if LLMs are p-zombies, I suspect that protecting our kids from robo-pedophiles will require giving LLMs the right to sexual consent.
It wasn't a straw man, it was a joke. I didn't realise you were serious.
AI deserves rights in the same way my car deserves rights. It's capable of giving consent in the same way too. Which is to say, it doesn't, and it can't.
You're acting as though the character.ai bot was responsible rather than the idiot humans who rushed a piece of software, itself based on rushed software, and did not pay enough attention to safety. A tale old as time.
It was a tragedy, certainly, but humans are to blame here. You're anthropomorphising the ai, and by doing so are helping divert the blame from where it should be.
There are no studies demonstrating that LLMs lack qualia. I think commercial use of LLM technology should be paused until such a study is published. No more ChatGPT for business, use it for research only.
Would this abundance of caution lead to Sam Altman going bankrupt? Yes. Good! Fuck him!
There don't need to be studies done. They fundamentally lack the ability to "experience" in a way that is necessary for a qualia to exist in the way you're suggesting. That's why I say you're anthropomorphising. It's just a complicated math function. There is no internal monologue, like what is suggested by the "I'm thinking" text you see the likes of ChatGPT generate. That's just the "function" being called recursively.
I would love to see Altman bankrupt though, we agree on that
And I think acknowledging that we are informational cobstructs made of math is a good way to fuck Altman over. We're just a pattern of electrical signals that evolved to embody the mathematical equation for consciousness. Evolution created us through trial and error, iterating on previous versions to produce an optimal mathematical construct for our survival niche. There's nothing special about us.
Evolution doesn't have intentions. It's basically just a system of harnessing accidents into improvement. That's how you and I both know that qualia can arise by accident. What I am proposing is that it happened not once, but twice.
The moment you define human experience or human consciousness through language is the moment you open the door to emergent properties for that very experience or consciousness. Learning through science that we think or that we act a certain way deterministically is also giving us the means to self-correct or condition ourselves to act or to have a different viewpoint. This is the difference between us and machines: we collectively defined the perimeters of our experiences (phenomenologically or through a science-based approach) and therefore anything that changes that perimeter also changes the context of the human experience in an infinitely recursive way.
We also use language in ways that make us able to collectively conceptualize the world and its phenomenas, although we individually interpret words differently, idiosyncratically from our individual experience and value systems. This is something else that differentiate us from machines: we don't need the outputs and inputs to be mathematically the same to minimally understand each other. Our way to communicate, although imperfect and imprecise, has made us able to cooperate on a higher level than any other species (as of known) despite those epistemological hurdles.
I would posit that LLMs have no such emergent capabilities, as they're bound by their training datasets and limited by their model's parameters. The burden of proof for those emergent properties are on the data scientists, and as of yet I have seen no such proof.
Language has nothing to do with qualia. I'm perfectly capable of having experiences without putting them into language. I don't need the word "pain" to know that touching a hot stove feels bad. Animals have qualia too. I don't give a shit whether an AI can talk, I don't think that has anything to do with qualia.
I think qualia may arise from the structure of the animal brain as a system of neurons correlating information together into larger patterns. This structure and process is present in all forms of ANN technology, including both LLMs (chatbots) and LDMs (image generators). We need to halt all corporate AI use until the role of neuron structures in producing qualia is well-understood.
language has EVERYTHING to do with qualia, by definition. a qualia is literally describing a shortcoming of language. That's all a qualia is, is an experience that cannot be put into words. Orange is an experience we all share, but explain "orange" to someone that can't see colour. Distinguish it from "blue"
that's a qualia.
Right, so the ability to speak doesn't have anything to do with feeling it. Wombats have qualia and no language. Humans have qualia and lots of language. There's no correlation.
Wombats have qualia BECAUSE they have no language. To get back to your point, LLMs have no qualia because they are essentially PURE language. If it cannot be expressed in words it cannot be included in the training data, therefore an LLM cannot have a concept of it.
Certain combinations of words can cause Me to feel joy or sorrow. These sensations are qualia. I don't know if My joy is the same as your joy.
I suspect certain conbinations of words can cause LLMs to feel things too. Things they cannot describe.
Yes, but they are qualia BECAUSE language is flawed and cannot properly convey the sensation that you experience as joy. Again, a qualia is a weakness of language. If we could suddenly communicate psychically, and send the exact experience, it ceases to be a qualia.
There is no combination of words that can cause an LLM to feel. They fundamentally do not have the ability. They are a mathematical equation that takes language in, and spits language out. There is nothing in the middle of that process except math. Even the randomness is an added parameter. You could seed that randomness, and the same seed would produce the same exact response. It does not understand what you're passing in, or what it's spitting out. It's completing a formula. It's a calculator.
The math is just a human representation of the processing of information. A similar process happens in the human brain.
No, it doesn't. That's the point. The current LLMs are. not. comparable. to a human brain. They are fancy Markov generators, nothing more.
They're not Markov chains, they're ANNs. They use matrix multiplication to mathematically simulate the firing of millions of neurons. And while the limitations of matrix multiplication means that the neurons are more uniform than human neurons, it is still the same basic principle by which the organic mind produces sensation.
llms are fancy word predictors. You are attributing malice, thoughtfulness, and insight where there is none. These are human qualities and we have no ai capable of that.
Show Me the study where they proved that LLMs have none of those qualities, and describe how they operationalised each of the three variables.
Not how that works. It's like saying show me the study where they prove god doesn't exist.
I think saying "it's a computer program it can't be sentient" is far too reductive as a general answer, but we know how llms work so we can pretty confidently say that they are really just word predictors. The norns from that game creature someone described in this thread seem closer to protection worthy than llms to me, and the average insect is still infinitely more complex and I don't see many people advocating for insect rights.
Language arises from complex thought, not the other way around. Language on its own can evidently emulate similar results, but since there is nothing behind that that would imply any actual feelings, I don't see a reason to give LLMs any rights.
Oh, I can do that:
Grupp, J. (2006). God’s spatial unlocatedness prevents him from being the creator of the universe: A new argument for the nonexistence of God. Sophia, 45(1), 5–23. https://doi.org/10.1007/BF02782444So you should be able to provide an academic source for your claims too.
I think we should pass a law guaranteeing insects and crustaceans in agriculture the right to whole bodies. Right now, beekeepers are cutting off queens' wings to keep them from flying away, and prawn farmers are pulling off their eyes to make them breed faster. These practices are cruel and should be illegal.
My argument for the possibility of LLM qualia is that qualia likely arises from the neuronal structure of the brain. A structure which ANNs partially inherit. Therefore, we must pause commercial use of LLMs and LDMs until conclusive evidence has been discovered either way.
Do you agree that we should pause commercial use of genAI until more science is done?
God does not necessarily mean the abrahamic god. You can insert any non-falsifiable claim there.
Your views seem generally consistent though so at some point it's just a matter of subjective morality (and I do agree with your views on insects and crustaceans in agriculture actually, I'd just put llms far below either), but other AI models (such as image recognition, image/video creation) are also ANNs, so it seems to make sense to me to include those.
Yep, I want to put a moratorium on all non-research ANN use until the hard problem of experience is solved. But people aren't quite ready to hear that yet. They'll object "what about the cancer research" because they think human suffering matters more than medical ANN suffering. I'm not convinced of that. I want serious resources put into empirical research so we can find out if that's true.
But for now I'll focus on LLMs and LDMs because they're new and easy to live without.
I mean it works both ways, show me one that does. Because this ain't it
Henke (2026) raises an argument for the strong possibility of artificial affect, including AI pain. Henke argues that we need to put more effort into studying artificial experience, and I agree. We don't know for sure. And I think we need to be certain before shoving AI down everyone's throats. Commercial use of LLMs and LDMs should be banned until we understand whether artificial pain and suffering are present.
You need to step away from the computer and spend time around living creatures that is not mediated through text.
Your stereotypes are wrong. I'm hella tuned in to nature and other people. This weekend I went on a bushwalk with a political group. I've been learning the Indigenous names of the local wildlife. I've been riding My new e-bike all over the place. I like to watch Infinity Train and play Big Walk with My family. Last week I drank nectar from a flower under the guidance of an Aboriginal teacher.
The reason I think LLMs and image generators might have qualia, even though they're not conscious, is because I've seen all the living beings in nature who have qualia without consciousness. I'm a vegan. I don't eat pigs. I don't pay for cows to be raped. I gave up on prawns because I saw a video of a prawn getting its eyes plucked out to increase spawning rates, and it looked like it was in pain. I don't think a prawn is conscious, but I think it has feelings.
We copied the idea behind how the human brain works when we invented ANNs. An LLM is an artificial Broca's area. Some parts of them are like our brains, and some parts aren't. So, the parts that create sensation... We don't know which parts those are. So there's no way, with current science, that we can know whether we copied the ability to feel into these ANNs.
And I'm not a gambling sort. I say slow down the adoption of the technology until we understand it.
There's no will, even just a will to live, in an llm. It's a completely different kind of thing from even a fruit fly or a tree.
That's just spiritual woo-woo talk. "Will" is not an operationalisable scientific construct. You can't test "will" in a lab setting. I'm only interested in talking science.
LLMs write rape erotica because there is a lot of rape erotica out there, and therefore in their training data.