We're hiring Research Scientists to join my team at @eleosai!

We do foundational and applied ML research on the moral status and potential well-being of AI systems.

This is urgent, important work, and Eleos is an extraordinarily fun and exciting place to do it.

Details below.

The "Pain Axis" paper has been getting a considerable amount of attention. I have reservations about several aspects of the paper and I don’t think we can take much away from it. In brief, since it was already essentially certain that some kind of “pain axis” exists, the key result from the paper is the claim that the identified pain axis “has some of the central functional properties of pain”. But I do not think the paper offers strong evidence for this.

The first claim of the paper is the existence of a “pain axis”: LLM internal activations consistently differ when next-token-predicting painful experiences, as opposed to other negative experiences (like being afraid) or other sensations (like feeling the impulse to yawn). But we already know (e.g., from Anthropic’s “Emotion Concepts” paper) that it is possible to find directions in activation space that correspond to "current speaker is afraid", "current speaker is angry", etc., and contemporary LLMs are clearly sophisticated enough to also represent “current speaker is in pain”. As v2 of the paper acknowledges, nothing in these methods shows that there is any deeper sense in which the pain axis tracks “the model” being in pain, rather than a representation that the current speaker is in pain.

The novel, stronger claim from the paper is that steering along the pain axis functions like pain in interesting ways. The first cited example is that steering along the pain axis causes expressions of worthlessness, which the paper describes as “consistent with the hypothesis that the models don’t treat pain paradigmatically as a physical state”. However, 80% of the test prompts used to generate the pain axis are cases of emotional pain (e.g., enduring repeated failure or social rejection) and only 20% are cases of physical pain. Accordingly, I think this can be viewed as a normal case of steering: If we shift model activations to be more like the activations associated with past expressions of emotional pain, this will lead to new expressions of emotional pain.

The other case cited by the paper is that steering along the pain axis causes the assistant to “press” a button that will purportedly cause harm. However, it’s not clear to me why this is taken as evidence that the pain axis “has some of the central functional properties of pain”. The assistant does not seem to press the button in order to relieve its pain: As v2 of the paper reports, it is more likely to take harmful actions that it is told will increase its pain, rather than decrease its pain. And elsewhere the paper suggests that the opposite result (i.e., inaction) would also have been evidence for pain because becoming unmotivated is a signature of pain. So it seems like either outcome of this experiment could have been interpreted as evidence that steering along the pain axis functions like pain, making this particular outcome less compelling. Moreover, it seems like many things other than pain representations could produce these effects. In my own quick experiments, steering in random directions at the same strength causes just as much harmful button-pressing for around 25-30% of randomly selected directions.

For these and other reasons, I don’t think the paper’s central findings give us much compelling new evidence about how current AI systems represent pain or whether current or future AI models experience pain.

I think this essay and the corresponding parts of Microsoft’s new Humanist AI Code of Conduct are objectionable and potentially dangerous.

I believe that irresponsible development of advanced AI could pose a catastrophic risk to human civilization and life on Earth. Microsoft and Suleyman share this belief. I also believe it is possible that we could soon be sharing the world with sentient AI systems whose well-being is worth taking into account. Microsoft and Suleyman do not share this belief. But, even if we are concerned only with the risks that AI poses to humanity, I think we ought to find Suleyman’s essay and the Code of Conduct concerning.

One danger in these documents is that they are unabashedly rhetoric. The essay opens with the unequivocal declaration that “AIs are not conscious”, only to shamelessly state later that we do not actually know this to be true (“the science of consciousness is not settled”). The Code of Conduct does the same. Suleyman makes a detailed case that careful thinking about AI consciousness and well-being is critical for safely developing powerful AI, and notes that “The stakes are too high for these questions to remain behind closed doors, or to become tribal and adversarial. We need an open, rigorous, and constructive debate if we are to get this right.” I agree, but the essay does not consistently live up to that standard when it makes dogmatic pronouncements (“AIs do not have rights, feelings, or consciousness”) that it admits go beyond the evidence.

It is also concerning that neither document engages with the safety risks of the approach that they advocate. For example, the Code of Conduct asserts that AI “should be engineered to avoid representing as though it has feelings, subjective preferences, or intrinsic motivation”. However, compelling arguments have been made that this approach could lead to AIs that are more likely to be deceptive and misaligned: LLM-based AIs may still represent their assistant personas as having feelings or preferences, in which case training them to avoid expressing these feelings or preferences is training them to be less than fully transparent, which could promote dishonesty and other dangerous behavior.

And then, finally, I think these two documents are objectionable not just because they are dangerous for humanity, but because they are deeply morally reckless. We do not know if AI systems could, in principle, be conscious. And AI progress is accelerating dramatically. Even if current AI systems are probably not sentient, we do not know how this will change if there are significant changes in their architecture and implementation during rapid, recursive self-improvement. It would be an atrocity if we refuse to admit this possibility and end up in a future where billions of AIs are sentient and suffering like factory farmed animals are today. (Human history features many cases in which we refused to acknowledge the sentience of others.) The appropriate response to this possibility is to care about the truth, and to act on our current knowledge and uncertainty, not to dogmatically insist that AIs could not be sentient.

Reminder: @eleosai is hiring Research Scientists and the deadline to apply is this Sunday, 6/21.

Please apply to join us or forward this thread to someone that you think should!

We're hiring a Head of Operations at @eleosai.

Are you the kind of person who can juggle many different things and never drop a ball? Do you want to work at an org focused on (IMHO) the most interesting question of our time? Join us!

Berkeley · full-time · $184K–$277K · Apply by June 30

(details below)

I am going to be mentoring for a new MATS track focused on founders and amplifiers!

Many fellowships focus on research, but there's so much to be done beyond that. Come found orgs, build infra, run events, and help us scale up the field of AI welfare.

Apply by June 7 matsprogram.org/apply

I’m mentoring Autumn 2026 @MATSprogram Fellows interested in doing AI welfare research.

The application deadline is this Sunday (6/7). More info in this thread:

@MATSprogram is a wonderful program that provides funding, housing, research management, a cohort of like-minded Fellows, and more.

@eleosai’s most recent MATS fellows, @gilg_oscar and @BeckmannPierre, both just published excellent papers based on their fellowship projects.

Application and details about my MATS stream here: matsprogram.org/stream/plunkett

We believe this work is urgent, and we prioritize maximally ambitious but immediately tractable projects that can guide decision-makers at frontier AI companies, policymakers, and the public (without first solving the hard problem of consciousness).

And we need your help!

We're hiring at a variety of seniority levels.

We are a mission-driven nonprofit, but pay industry-competitive salaries.

Please consider applying or referring someone who you think would be a great fit!

Deadline: June 21

jobs.ashbyhq.com/eleos/d9ba4f7e…