You don't have to crack it to make it but you have to crack it to determine whether you've made it. That's kinda the trick of the early AI hype, notably that NYT article that fed Chat GPT some simple sci fi, ai-coming-to-life prompts and it generated replies based on its training data - or, if you believe the nyt author, it came to life.

I think what you're saying is a kind of "can't define it but I know it when I see it" idea, and that's valid, for sure. I think you're right that we don't need to understand it to make it - I guess what I was trying to say was, if it's so complex that we can't understand it in ourselves, I doubt we're going to be able to develop the complexity required to make it.

And I don't think that the inability to know what has happened in an AI training algorithm is evidence that we can create a sentient being.

That said, our understanding of consciousness is so nascient that we might just be so wrong about it that we're looking in the wrong place, or for the wrong thing.

We may understand it so badly that the truth is the opposite of what I'm saying : people have said ("people have said" is a super red flag, but I mean spiritualists and crackpots, my favorite being the person who wrote The Secret Life of Plants) that consciousness is all around us, that every organized matter has consciousness. Trees, for example - but not just trees, also the parts of a tree; a branch, a leaf; a whole tree may have a separate consciousness from its leaves - or, and this is what always blows my mind: every cell in the tree except one. And every cell in the tree except two, and then every cell in the tree except a different two. And so on. With no way to communicate with them, how would a tree be aware of the consciousness of it's leaves?

How could we possibly know if our liver is conscious? Or our countertop, or the grass in the park nearby?

While that's obviously just thought experiment bullshit, my point is, we don't know fucking anything. So maybe we created it already. Maybe we will create it but we will never be able to know whether we've created it.

Well, if you experience consciousness, that's what consciousness is. As in, the word and concept "consciousness" means being conscious, the way you experience being conscious right now (unless of course you're unconscious as I write this...). Free will does not enter into it at the basic level, nothing says you're not conscious if you do not have free will. So what would it really mean to say consciousness is an illusion? Who and what is having the illusion? Ironically, your statement assumes the existence of a higher form of consciousness that is not illusory (which may very well exist but how would we ever know?). Simply because a fake something presupposes a real something that the fake thing is not.

So let's say we could be certain that consciousness purely is the product of material processes in the brain. You still experience consciousness, that does not make it illusory. Perhaps this seems like I'm arguing semantics, but the important takeaway is rather that these kinds of arguments invariably fall apart under scrutiny. Consciousness is actually the only thing we can be absolutely certain exists; in this, Descartes was right.

So, it's meaningful to say that a language model could "fake" consciousness - trick us into believing it is an "experiencing entity" (or whatever your definition would be) by giving convincing answers in a conversation - but not really meaningful to say that actual conscious beings somehow fake consciousness. Or, that "their brains" (somehow suddenly acting apart from the entity) trick them.

I like this take - I read the refutation in the replies and I get that point, but consciousness as an illusion to rationalize stimulus response makes a lot of sense - especially because the reach of consciousness's control is much more limited than it thinks it is. Literally copium.

When I was a teenager I read an Appleseed manga and it mentioned a tenet of Buddhism that I'll never forget - though I've forgotten the name of the idea (and I've never heard anyone mention it in any other context, and while I'm not a Buddhist scholar, I have read a decent amount of Buddhist stuff)

There's some concept in Japanese Buddhism that says that, while reality may be an illusion, the fact that we can agree on it, means that we can at least call it "real"

(Aka Japanese Buddhist describes copium)