this paper answers a question that has been bothering me for a while: how much can we expect to get from rubrics vs the tacit knowledge of humans?
🚨New work: A SOTA prompt optimization framework + a novel unsupervised measure of prompt performance.
Written rubrics and constitutions are crucially important for AI task improvement, safety following, pluralism and adaptation.
But some of the things we might want a rubric put into words; how would you even describe “comic voice”? I’m human, albeit not that funny, and I'm totally not sure what that means.
What kinds of written constructs can you expect your judge to do well on and how good can you expect it to be?
0
1
43
3,597
did i already mention how the @humansand team is cooking?
We accelerated one of the most optimized NVFP4 decode kernels while also making it more accurate
Fewer bits typically mean more speed but less accuracy. We did the opposite: more bits, but up to 10% faster generation
Already used in Ampersand’s RL and Boba serving
0
1
31
2,063
first we built Persimmon, our sota human simulator. using that we trained Ampersand, a model that understands people individually and in groups. which powers Boba — released today! @humansand is cooking!
Introducing Boba - the social intelligence that understands your goals and your community. Text Boba today or message it online at boba.app
6
7
85
4,047
noahdgoodman retweeted
Very excited to share what we have been working on at Recursive! A huge congrats to our entire team. We’re excited to hear what you all think. 🧪 🧬 🔭 🚀 📈✨
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from hs
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
6
5
88
5,719
while evaluating Ampersand v0.1 for sycophancy @__Charlie_G found something interesting: deployment makes models more sycophantic, but good human simulators substantially reduce evaluation awareness.🧵
1
3
53
3,135
but good user simulators in evaluations radically reduce eval awareness. the persimmon v0.1 user model with test-time scaling makes a huge difference. for Argon P(deploy) goes from 20% to 82% in our evals!
1
0
5
256
more details in the Ampersand v0.1 model card: ampersand.humansand.ai/blog/model-car…
0
0
6
214
we've been developing a substantially different model training paradigm at @humansand, focussed on rich social simulation over long term outcomes. ampersand is our first model with this training, released today!
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable ls
Available to everyone through our API
5
4
113
4,964
i'm proud of the excellent @humansand team, and how hard they've worked to pull this together. this is a first step and we've got Big Plans...
1
2
20
1,136
i've made some changes -- thanks for comments! here is the interactive version if you want to play with it: ngoodman.github.io/ai-safety-map
i'm working on an AI safety lecture for my class, Minds and Machines. here is the first pass on a conceptual map... i would love suggestions from my safety friends.
2
2
47
3,638
i'm working on an AI safety lecture for my class, Minds and Machines. here is the first pass on a conceptual map... i would love suggestions from my safety friends.
12
17
155
14,910
@ancadianadragan @rohinmshah @AllanDafoe @sleepinyourhat @RogerGrosse @EthanJPerez @OwainEvans_UK (and many others, but i'm rushing)
2
0
2
1,262
i find this somewhat satisfying, but what i'm really waiting for is biting commentary by my friend @GaryMarcus. every time he attacks me (or hires me, we've done both) i know i'm on to something...
How to build a billion-dollar-valued AI research lab in 2026:
1. Share your critical take on huge companies like @humansand ($8.5B).
2. People who invested in them, or who attached their names to them (@Stanford, @OpenAI, @CarnegieMellon , etc.), will show up in the comments.
3. Congratulations. By being compared to them, your company is now also valued as a BILLION-DOLLAR COMPANY.
😂 Just joking.
Btw, at @walnutinfra we’re about to release some pretty cool research soon.
Follow us and stay tuned
1
0
31
8,517
RT @alexisjross: both of these tweets highlight the challenges of coming up with good evals for user models.
1) the ELIZA case study highli…
1) the ELIZA case study highli…
0
1
0
75
i find this especially amusing because i'm teaching ELIZA in Minds and Machines (Stanford's intro cogci w/ @mcxfrank) tomorrow...
People often eval user models on the "User Sim Index" - we probably shouldn't. I attached code for a "user model" scoring 95% on its 4 behavioral dims: 97% comms, 97% info, 92% clarify, 95% error reaction. I think this beats any released model. See also a sampled conversation
1
3
34
5,986