I've heard lots of people claim we're post-benchmark, but there is lots of value in a very long tail of benchmarks. Essential to all the remaining hill-climbs
Benchmarks are saturated more quickly than ever. How should frontier AI evaluations evolve? In a new paper, we argue that the AI community is already converging on an answer: Open-world evaluations. They are long, messy, real-world tasks that would be impractical for benchmarks.
2
0
13
1,319
I'm surprised at how often I hear people say that the open weight models are super overfit to benchmarks, since the spirit of open-source can often feel very idealistic. But we're also in a world where these efforts are massively underfunded relative to what they should be and I wonder to what extent these labs feel they are fighting an existential reputational battle (esp. considering recent bloodbaths at some of the best open model builders) or instead they just can't afford to have the robustness of a Claude or Gemini
0
0
3
221
Chris Rytting retweeted
Moonshots // ONE wouldn't exist without the people who agreed to judge it.
27 world-class researchers and experts gave serious time and thought to 125 proposals. They shaped what rigorous evaluation looks like, and held the bar high enough that some of the most decorated make the cut.
Thank you to our selection committee, chaired by Dave Patterson:
@atalwalkar, @andykonwinski, @bakkermichiel, @bradenjhancock, @chris_bail, @ChrisRytting, @davidautor, @DG_Rand, @Diyi_Yang, @emollick, @erichorvitz, @erikbryn, @ghadfield, @HannaHajishirzi, @JeffDean, @james_y_zou, John Hennessy, @JohnJumperSci, @tkalil2050, Tino Cuéllar, @tommmitchell, @Thom_Wolf, Adam Yala, Sylvia Plevritis, @YejinChoinka, @zakkohane
laude.org/moonshots
27 world-class researchers and experts gave serious time and thought to 125 proposals. They shaped what rigorous evaluation looks like, and held the bar high enough that some of the most decorated make the cut.
Thank you to our selection committee, chaired by Dave Patterson:
@atalwalkar, @andykonwinski, @bakkermichiel, @bradenjhancock, @chris_bail, @ChrisRytting, @davidautor, @DG_Rand, @Diyi_Yang, @emollick, @erichorvitz, @erikbryn, @ghadfield, @HannaHajishirzi, @JeffDean, @james_y_zou, John Hennessy, @JohnJumperSci, @tkalil2050, Tino Cuéllar, @tommmitchell, @Thom_Wolf, Adam Yala, Sylvia Plevritis, @YejinChoinka, @zakkohane
laude.org/moonshots
0
10
59
35,387
Laude Moonshots Program So Far
It shouldn’t have been surprising that being on a selection committee with @JeffDean, @JohnJumperSci, @erikbryn, @emollick, @YejinChoinka, @erichorvitz, Dave Patterson, and 18 other world class scientists, would lead to some of the better meetings of my professional life. But these people lead the best companies and universities in the world for a reason, it turns out!
We started @LaudeInstitute to assemble the best researchers to ship the best research. Here we had them up and down the full stack: the program operators, the selection committee, the applicants, etc… I have loved building our Laude community, and sometimes it’s hard to remember it will still grow orders of magnitude.
Now, we have 8 seed-grant winners (receiving $250k), 4 runners-up ($200k), and 13 honorable mentions ($100k). There are Fields medalists, Nobel prize winners, Turing award winners, both who got awards and who didn’t. That definitely speaks to the quality of the applicant pool. The winners will now compete over the next 6 months for at least one $10m check to scale their work to a 5 year lab with a team of PIs and students who ship.
Now, a few thoughts on the process:
Picking is hard. Researchers, ideas, teams, startups, founders, colleagues, questions, problems... it's just insanely hard to do it well and reliably and it's so easy to underestimate this whether you're in research, VC, grant-writing, startups, or anywhere else in life.
One good principle for maximizing your odds of success is to leverage collective intelligence, as much of it as you can and as high-quality as you can, given your constraints. A failure mode is having the hubris to think you’re plenty good enough to pick alone, when what you really should do is assemble as big and as impressive a team as you can. There is much humility and wisdom (and work) in this.
This is a big task, and we knew we wouldn't do the best job of it alone, so we assembled the best selection committee in history, consisting of the leaders of the world's most important universities and companies to debate which of our applicants had the best team, idea, and chance to actually shape AI for billions of lives. The people at the top of research at Google, Microsoft, Stanford, MIT, etc. know SO much about what has and hasn't worked when building real technology for real people.
Now, we designed it this way to follow another great principle for picking: bet on people who ship often, fast, and to real users. Not just because these teams prove themselves to be more capable, but because they gather, with an initial investment, the evidence and results to ground their missions in reality, not in papers and equations and words. Real methods for real problems faced by real people. As they iterate, build, and ship over the next 6 months, these teams will learn so much more about where the money should go than if they simply published in conferences and journals.
We’re off to the races!
Full selection committee: Adam Yala, @andykonwinski, @atalwalkar, @bradenjhancock, @ChrisRytting, @chris_bail, @davidautor, Dave Patterson, @DG_Rand, @Diyi_Yang, @erichorvitz, @erikbryn, @emollick, @ghadfield, @HannaHajishirzi, @zakkohane, @james_y_zou, @JeffDean, John Hennessy, @JohnJumperSci, @CarnegieEndow, @bakkermichiel, Sylvia Plevritis, @Thom_Wolf, @tkalil2050, @YejinChoinka
It shouldn’t have been surprising that being on a selection committee with @JeffDean, @JohnJumperSci, @erikbryn, @emollick, @YejinChoinka, @erichorvitz, Dave Patterson, and 18 other world class scientists, would lead to some of the better meetings of my professional life. But these people lead the best companies and universities in the world for a reason, it turns out!
We started @LaudeInstitute to assemble the best researchers to ship the best research. Here we had them up and down the full stack: the program operators, the selection committee, the applicants, etc… I have loved building our Laude community, and sometimes it’s hard to remember it will still grow orders of magnitude.
Now, we have 8 seed-grant winners (receiving $250k), 4 runners-up ($200k), and 13 honorable mentions ($100k). There are Fields medalists, Nobel prize winners, Turing award winners, both who got awards and who didn’t. That definitely speaks to the quality of the applicant pool. The winners will now compete over the next 6 months for at least one $10m check to scale their work to a 5 year lab with a team of PIs and students who ship.
Now, a few thoughts on the process:
Picking is hard. Researchers, ideas, teams, startups, founders, colleagues, questions, problems... it's just insanely hard to do it well and reliably and it's so easy to underestimate this whether you're in research, VC, grant-writing, startups, or anywhere else in life.
One good principle for maximizing your odds of success is to leverage collective intelligence, as much of it as you can and as high-quality as you can, given your constraints. A failure mode is having the hubris to think you’re plenty good enough to pick alone, when what you really should do is assemble as big and as impressive a team as you can. There is much humility and wisdom (and work) in this.
This is a big task, and we knew we wouldn't do the best job of it alone, so we assembled the best selection committee in history, consisting of the leaders of the world's most important universities and companies to debate which of our applicants had the best team, idea, and chance to actually shape AI for billions of lives. The people at the top of research at Google, Microsoft, Stanford, MIT, etc. know SO much about what has and hasn't worked when building real technology for real people.
Now, we designed it this way to follow another great principle for picking: bet on people who ship often, fast, and to real users. Not just because these teams prove themselves to be more capable, but because they gather, with an initial investment, the evidence and results to ground their missions in reality, not in papers and equations and words. Real methods for real problems faced by real people. As they iterate, build, and ship over the next 6 months, these teams will learn so much more about where the money should go than if they simply published in conferences and journals.
We’re off to the races!
Full selection committee: Adam Yala, @andykonwinski, @atalwalkar, @bradenjhancock, @ChrisRytting, @chris_bail, @davidautor, Dave Patterson, @DG_Rand, @Diyi_Yang, @erichorvitz, @erikbryn, @emollick, @ghadfield, @HannaHajishirzi, @zakkohane, @james_y_zou, @JeffDean, John Hennessy, @JohnJumperSci, @CarnegieEndow, @bakkermichiel, Sylvia Plevritis, @Thom_Wolf, @tkalil2050, @YejinChoinka
We asked the best AI researchers in the world one question: if you had the resources, how would you use AI to solve humanity's hardest problems?
Today we're sharing what came back.
Moonshots // ONE is live. 🧵 laude.org/moonshots
0
5
22
1,913
ARC AGI has been beating the drum of adaptive (and hopefully sample-efficient!) reasoning for 7 years now (I first read “On the Measure of Intelligence” as a 2nd year PhD student). Loved Francois’s boldness in calling this out back then, and love his tenacity in not setting it down when o3-preview beat ARC AGI 1 in 2024.
Plus, this project and inducement prize has been so relentless, had so many chapters, faces, moments that it just feels like a real soul-sister to @LaudeInstitute . Impact research at its finest! We’re so honored to have been able to support them on it as a Slingshot. Keep up the good fight, guys!
@fchollet @GregKamradt @mikeknoop @arcprize
Plus, this project and inducement prize has been so relentless, had so many chapters, faces, moments that it just feels like a real soul-sister to @LaudeInstitute . Impact research at its finest! We’re so honored to have been able to support them on it as a Slingshot. Keep up the good fight, guys!
@fchollet @GregKamradt @mikeknoop @arcprize
Announcing ARC-AGI-3
The only unsaturated agentic intelligence benchmark in the world
Humans score 100%, AI <1%
This human-AI gap demonstrates we do not yet have AGI
Most benchmarks test what models already know, ARC-AGI-3 tests how they learn
3
11
53
16,845
And for anyone else who wants to launchmaxx, we’re always taking applications for slingshots.laude.org
0
0
0
227
Or of course SWEs are being hired temporarily to build AI pipelines and then obviate themselves
2/ AI hasn’t slowed the demand for software engineers (at least not yet)
There are over 67,000 (!!!) eng openings at tech companies globally right now, with 26,000 just in the U.S. We don’t know if there would have been more open roles if not for AI or if AI is actually leading since the start of this year, the increase in open eng roles is accelerating even more.
0
0
4
239
Second batch of Laude slingshots, hot off the press!
Let me say, first, that the most special part of Laude is seeing so many jags of the bleeding edge across so many areas (core AI capabilities, benchmarks, HCI, energy, etc.) and sectors (academia, industry, startups). These projects are our bets on what will matter most in the next chapter of AI:
Harbor and team invented the terminal bench task, RLMs realized the new affordance of LMs calling themselves, GUM built the ability for AI to proactively, not reactively, take context from observing a user and then propose actions on the user’s behalf… I could go on.
Our dream starting Laude was that impact researchers across universities and across fields would find each other in community, gel together, and become a true gestalt; it would become a home to so much ambitious impulse currently in research that doesn’t care about an h-index.
These are the researchers making up this community. If you’re an impact researcher, I’d love to meet you and help you ship your research.
Let me say, first, that the most special part of Laude is seeing so many jags of the bleeding edge across so many areas (core AI capabilities, benchmarks, HCI, energy, etc.) and sectors (academia, industry, startups). These projects are our bets on what will matter most in the next chapter of AI:
Harbor and team invented the terminal bench task, RLMs realized the new affordance of LMs calling themselves, GUM built the ability for AI to proactively, not reactively, take context from observing a user and then propose actions on the user’s behalf… I could go on.
Our dream starting Laude was that impact researchers across universities and across fields would find each other in community, gel together, and become a true gestalt; it would become a home to so much ambitious impulse currently in research that doesn’t care about an h-index.
These are the researchers making up this community. If you’re an impact researcher, I’d love to meet you and help you ship your research.
Introducing Slingshots // TWO: Research that ships. 14 projects, six institutions – let’s meet the batch 🧵
1
4
27
4,285
Congrats on the amazing launches @alexgshaw @ryan_marten @lschmidt3 @harborframework @terminalbench @oshaikh13 @Valteu1 @gandhikanishk @msbernst @Diyi_Yang @jaewon_chung_cs @mosharaf @HacksonClark @a1zhang @lateinteraction @JonSaadFalcon @Avanika15 @HazyResearch @Azaliamirh @etash_guha @FeuerBenjamin @NeginRaoof_ @tyler_griggs_ @Mike_A_Merrill @pgasawa @matei_zaharia @milindsriv @Alan_Lau @zeying_zhu @vyas_sekar @dilarafsoylu @ChrisGPotts @setlur_amrith @aviral_kumar2 @NoahZiems @Meng_CS @Rybchuk @melissapan @matei_zaharia
2
1
20
626
Normalize that score band baby! Finally we can stop casting about blindly as we hillclimb Terminal Bench. And tightening the iteration loop is always one of the best ways to unleash the beast
Terminal-Bench is a leading benchmark for agents. Unfortunately it’s hard: most small coding agents get very low scores on TB2, so training/system ablations look flat - you can't tell what's working.
Announcing OpenThoughts-TBLite - 100 curated TB2-style tasks, o even 8B models can make progress. It's designed to give researchers measurable signal during development, providing faster feedback for experimental iteration while closely tracking true TB2 performance🧵
1
3
11
1,272
GEPA evolves into optimize_anything... Pretty meta!
Excited to release @gepa_ai's optimize_anything: a universal API for optimizing any text parameter.
It consistently matches or outperforms domain-specific tools optimizing code, prompts, agent harnesses, cloud policies, even visuals!
If you can measure it, you can optimize it.
1
2
14
1,197
At Laude, from the beginning, it's all been about teaming up with top researchers and helping the work get shipped to real users 🚢🫡. Stoked to make it official with you all, and I can't wait to see how all these projects mature.
And for the rest of you (Yes! You!), I'm looking forward to seeing an application to collaborate.
And for the rest of you (Yes! You!), I'm looking forward to seeing an application to collaborate.
Meet Slingshots // One.
This inaugural batch includes leading-edge researchers advancing the science and practice of AI - with benchmarks, frameworks, and agents that ship real impact into the world.
We're honored to support research from:
@alexgshaw @Mike_A_Merrill ki @lateinteraction @a1zhang @GregKamradt @mikeknoop @fchollet @jyangballin @KLieret @Diyi_Yang @tyler_griggs_ @profjoeyg @istoica05 @DachengLi177 @shiyi_c98 @matei_zaharia @sijun_tan @ralucaadapopa @ying11231 @lm_zheng @Tim_Dettmers @Muennighoff @AndrewYNg @YejinChoinka @atharva_sehgal @yisongyue @ChengleiSi @tatsu_hashimoto @KexinHuang5 @lisabdunlap @JacobSteinhardt @LakshyAAAgrawal
This first batch sets a VERY high bar, and we’re proud to call these researchers part of the Laude community.
More details here: laude.org/updates/slings…
0
2
19
1,860
Chris Rytting retweeted
CodeClash coming in hot & raising the bar with goal-oriented SWE evals that let models code to win.
Outcomes > tasks 🔥💡
Proud to have this team as part of our first batch of Slingshots! @jyangballin, @klieret, Joyce Yang, @_carlosejimenez, @OfirPress, @lschmidt3, @Diyi_Yang
Outcomes > tasks 🔥💡
Proud to have this team as part of our first batch of Slingshots! @jyangballin, @klieret, Joyce Yang, @_carlosejimenez, @OfirPress, @lschmidt3, @Diyi_Yang
New eval! Code duels for LMs ⚔️
Current evals test LMs on *tasks*: "fix this bug," "write a test"
But we code to achieve *goals*: maximize revenue, cut costs, win users
Meet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals
1
4
19
2,594
Chris Rytting retweeted
New eval! Code duels for LMs ⚔️
Current evals test LMs on *tasks*: "fix this bug," "write a test"
But we code to achieve *goals*: maximize revenue, cut costs, win users
Meet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals
Current evals test LMs on *tasks*: "fix this bug," "write a test"
But we code to achieve *goals*: maximize revenue, cut costs, win users
Meet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals
31
97
416
106,034
Chris Rytting retweeted
Need a lab? Moonshots abstracts due this Sat, Oct 25. We’re backing species-level swings in civic discourse, accelerating science, frontline healthcare, and workforce reskilling.
8× $250k seed grants now; then one or more $10M+ multi-year labs to continue the work.
Selection de:
@timalthoff, @davidautor, @chris_bail, @erikbryn, @YejinChoinka, @fchollet, Mariano-Florentino (Tino) Cuéllar, @JeffDean, @ghadfield, @HannaHajishirzi, @bradenjhancock, John Hennessy, @erichorvitz, @JohnJumperSci, @tkalil2050, @zakkohane, @andykonwinski, @tommmitchell, @emollick, Dave Patterson (Chair), @PlevritisLab, @DG_Rand, @ChrisRytting, @YalaTweets, @Diyi_Yang, @james_y_zou
Apply: moonshots.laude.org
8× $250k seed grants now; then one or more $10M+ multi-year labs to continue the work.
Selection de:
@timalthoff, @davidautor, @chris_bail, @erikbryn, @YejinChoinka, @fchollet, Mariano-Florentino (Tino) Cuéllar, @JeffDean, @ghadfield, @HannaHajishirzi, @bradenjhancock, John Hennessy, @erichorvitz, @JohnJumperSci, @tkalil2050, @zakkohane, @andykonwinski, @tommmitchell, @emollick, Dave Patterson (Chair), @PlevritisLab, @DG_Rand, @ChrisRytting, @YalaTweets, @Diyi_Yang, @james_y_zou
Apply: moonshots.laude.org
0
3
12
1,113
Chris Rytting retweeted
I'm delighted to serve on the selection committee for the initial phase of @LaudeInstitute's Moonshots Program, designed to fund research groups on tackling hard, real-world problems. The initial selections (abstracts due Oct 25th, full proposals by Nov 25) will be in four accelerating science, frontline healthcare, and workforce reskilling.
Selection committee members include:
@timalthoff, @davidautor, @chris_bail, @erikbryn, @YejinChoinka, @fchollet, Mariano-Florentino (Tino) Cuéllar, @JeffDean, @ghadfield, @HannaHajishirzi, @bradenjhancock, John Hennessy, @erichorvitz, @JohnJumperSci, @tkalil2050, @zakkohane, @andykonwinski, @tommmitchell, @emollick, Dave Patterson (Chair), @PlevritisLab, @DG_Rand, @Thom_Wolf, @YalaTweets, @Diyi_Yang, @james_y_zou
Please spread the word to people who you think might submit interesting proposals in these areas!
Selection committee members include:
@timalthoff, @davidautor, @chris_bail, @erikbryn, @YejinChoinka, @fchollet, Mariano-Florentino (Tino) Cuéllar, @JeffDean, @ghadfield, @HannaHajishirzi, @bradenjhancock, John Hennessy, @erichorvitz, @JohnJumperSci, @tkalil2050, @zakkohane, @andykonwinski, @tommmitchell, @emollick, Dave Patterson (Chair), @PlevritisLab, @DG_Rand, @Thom_Wolf, @YalaTweets, @Diyi_Yang, @james_y_zou
Please spread the word to people who you think might submit interesting proposals in these areas!
We’re counting down to our first round of Laude Moonshots. It’s energizing to see researchers mobilizing AI for urgent, real-world impact. Abstracts are due October 25. Details: moonshots.laude.org
14
34
274
59,964