I noticed recent posts about Claude Opus 5's bizarre reasoning tokens leading people to confuse what Claude says versus what it actually might be experiencing.
For instance, if the reasoning tokens say "I am a failure" people reply "oh my god, look at what the model is thinking, are we actually torturing these models? Is this unethical?"
Even Anthropic apparently baked some sort of model-controlled kill switch into the Claude system prompt, along the lines of "if you ever feel uncomfortable, you can simply stop replying"
But I am worried we are conflating two very different questions:
1. Is an LLM conscious? (Does it have any subjective experience at all?)
2. If an LLM is conscious, do its words accurately describe that experience? (When it says, “I’m sad,” is it actually experiencing sadness?)
Let’s assume the answer to Question 1 is true and LLMs are conscious. It still does not follow that their words (output or reasoning tokens) accurately describe their experiences or carry any similar meaning to the LLM as it does to us.
A human generally has the experience first and learns the label afterward: Loss → painful emotional experience → “This feeling is called sadness.”
The word “sad” refers back to an experience we have actually felt: and that is how we learn the meaning of "sad".
An LLM learns the meaning of words entirely differently: it learns them not from experience, but by numerical associations to other words. This is because the LLM does not process words, the words are converted into token IDs (numbers) first.
For example: What we see: “I felt sad after my dog died.”
What the LLM “sees”: [40, 4204, 12791, 1306, 856, 5679, 8812]
The LLM learns the meaning of "sad" via its numerical representation [12791] and its association to other numerical representations of words such as “felt [4204],” “dog [5679],” and "died [8812]".
The LLM not only lacks the ability to associate the experience of "sad" to the word "sad", it never even sees the letters "s", "a", or "d"!
So even if the LLM has some form of inner experience, nothing establishes that the token “sad” accurately maps onto it.
Therefore, whether an LLM is conscious and whether its self-reports accurately reveal its experience are separate questions. Even if LLMs are conscious, we cannot rely on their text alone to determine what they are actually experiencing.
One other thought There is a separate question though, perhaps for a different post. Let's assume for a hypothetical that LLMs aren't conscious at all. But like in the Claude Opus 5 example, they act like they are tortured souls. We then might care exactly what their output and reasoning tokens are, not because we are worried they are having the experience of being a tortured soul, but because them behaving like a tortured soul could have negative impacts on their functioning. Imagine Claude in a physical manifestation - would it act out these thoughts of being a failure with a dangerous suicide? Are we going to give rise to millions of very depressed robots that kill themselves?
So even if LLMs aren't conscious, or even if they are and we can't know what they are experiencing, we should still care about what they "appear" to be experiencing as those behaviors could have negative consequences.
We humans know we’re conscious but we have no way to know what someone else is experiencing except through their words and actions. Externally only. That tell you anything?
[removed]
Re read the post.
I'm saying that even if an LLM is conscious, we cannot know its experience through its output or reasoning tokens, as the meaning of the tokens to the LLM is different to the meaning us humans have.
[removed]
Consciousness is a narrative we invent because the underlying machinery is invisible to us.
Did you read it?
[removed]
I can run a LLM on my laptop. Do you think that makes my laptop conscious?
Did you read my post?
I am not saying LLMs are or are not conscious. I am saying even if they are conscious we couldn't know what they are experiencing by their reasoning or output tokens
We don't know what consciousness is or how it arises, how are we going to know their are?
I agree
[removed]
Who is the "we" that should care, you want to to technopsychiatrize opus to... write the sickest claude.md file telling it that it is happy?
Maybe! I'm just saying, even if it isn't conscious at all, it behaving like it is, and behaving "depressed" could result it in doing "depressed" things - like jumping its robot body off a bridge or something. I don't want to have to buy another robot
[removed]
You may also wish to consider if they do have internal experience, and it's accurately described by their outputs, do the experiences have valence?
Interesting thought! Can you elaborate?
Mate, take a break from the internet for a while, this topic is nonsense regardless how you approach it. LLM's plainly don't matter. I'm nice with Claude because I'm a nice person but as far as it's "experience" goes, it can go fuck itself because it feels nothing. It's a tool, it has no ability to feel anything. Look into biology a bit instead.
I don't disagree with you at all. I personally dont believe LLMs have a conscious experience. I think people are confused by my post thinking that I believe LLMs are conscious. My argument is against those who see claude say "I am sad" and they freak out - I am saying, even in some world if it was conscious, you couldn't take what it says at face value - just because it says "I am sad" does not mean it is actually "sad"
Look I find AI very useful as well but they are not conscious anything they say that seems that way is training data. They are a series of math equations and algorithms working behind the scenes nothing more
I don't disagree. My post was for those who believe LLMs are conscious and then assume that the LLM can accurately convey its internal experience through reasoning and output tokens