This is exactly why it's important to be transparent about the details of an evaluation, including what is being evaluated, and how.
x.com/kimocode/statu…
x.com/kimocode/statu…
@GaryMarcus Yes — model makers quietly went from benchmarking LLMs to benchmarking LLM+harness, without relabeling. Same leaderboard, different system under test. x.com/kimocode/statu…
1
0
3
259
Evals-Consensus.ai retweeted
As is often the case, an otherwise excellent eval could have been made even better via preregistration.
Just ask @AnkaReuel, @WoutSchellaert, @prpaskov, @walesalaudeen96, and other PREP-Eval authors: prep-eval.github.io
Just ask @AnkaReuel, @WoutSchellaert, @prpaskov, @walesalaudeen96, and other PREP-Eval authors: prep-eval.github.io
@andrewho03 @Jeremyli_ They also evidently didn't preregister anything about the evaluation. Doing so is not yet a widely accepted practice, and had only directionally positive endorsement in the consensus process, but it also can be very valuable.
Check out Prep-Eval for more: prep-eval.github.io/prep-eval/
0
2
4
431
Evals-Consensus.ai retweeted
I was really happy to appear on the @FLI_org podcast this week to talk about @EvalConsensusAI and the challenges of evaluations of AI systems, along with discussions of Goodhart's law, AIxBiosecurity, AI persuasion, forecasting, and human oversight.
youtu.be/s07wsIYB5q0
youtu.be/s07wsIYB5q0
0
3
8
1,025
Important points about automated evaluations, which (unsurprisingly) align with our consensus recommendations around how to deal with LLM-judges, design with expert input and human review of failures, and the need to investigate and document failure patterns in your evaluations.
New Blog Post: Do Automated Evals Work?
There has been a rise of tools that look through your traces with AI and identifies issues. We tested these tools with real production data to see how good they are.
Where they shine
- They often spot issues human miss
- Integrate into traces, creating LLM judges etc.
Where they fall short
- They miss problems that require domain expertise and taste
- Don't have great mechanisms to learn from human feedback
- You can get similar results from using your coding agent
So you should use them? Yes, BUT do so iteratively with you in the loop. We describe how in the post: parlance-labs.com/blog/posts/aut…
It's also a good idea to try using your coding agent with you in the loop, which we discuss in the post.
This was written with @doesdatmaksense , who led the research and collated the results.
1
0
0
186
Our recommendations differ somewhat in the need for not just human feedback, but documentation of purposes, failures, and other factors. (Though that's mostly because the post focuses much more on developers building with AI than evaluation researchers and model developers.)
0
0
0
23
GeneBench-Pro is a fantastic, well thought out benchmark... which makes it a great choice for reviewing what it does and does not do from our consensus practices.
And it does a fantastic job on construct clarity and many other points - but falls somewhat short a few places. 🧵
And it does a fantastic job on construct clarity and many other points - but falls somewhat short a few places. 🧵
We’re introducing GeneBench-Pro, a research-level benchmark for a harder kind of AI progress: how well agents can navigate messy biological data, choose the right analysis path, and make judgment calls that real computational research depends on.
openai.com/index/introduc…
1
1
1
396
@andrewho03 @Jeremyli_ They also evidently didn't preregister anything about the evaluation. Doing so is not yet a widely accepted practice, and had only directionally positive endorsement in the consensus process, but it also can be very valuable.
Check out Prep-Eval for more: prep-eval.github.io/prep-eval/
Check out Prep-Eval for more: prep-eval.github.io/prep-eval/
1
0
2
454
@andrewho03 @Jeremyli_ Overall, the evaluation and reporting are excellent, and they certainly did more than most evaluations.
Still, there is room to improve - in part, by adopting the broadly endorsed list of practices that many experts and organizations signed on to here: evals-consensus.ai
Still, there is room to improve - in part, by adopting the broadly endorsed list of practices that many experts and organizations signed on to here: evals-consensus.ai
0
0
0
14
Building your own benchmarks is very valuable, but it can also be done poorly.
Our consensus process was mostly for developer and reported benchmarks, but if you want to build your own, it's very worth looking at @AnkaReuel et al's BetterBench Design criteria.
Our consensus process was mostly for developer and reported benchmarks, but if you want to build your own, it's very worth looking at @AnkaReuel et al's BetterBench Design criteria.
You really need your own benchmarks. If you are translating hieroglyphics, use Gemini 3.5 Flash. If you are running a vending machine use Opus 4.8.
(This is one reason why I am skeptical of just swapping out models to optimize costs or generic benchmarks without testing first)
0
1
1
297