people are underestimating how quickly AI economics are changing. coding agents are getting way better AND way cheaper.
on sudo L7, thinky's new Inkling 2 Small model gets within 2 points of Argon and Muse Wonder - which are 8-13x more expensive.
and it beats several models 3.8 flash, glm 5.3) that cost 3–7x more.
Congrats to @johnschulman2 @miramurati and the entire @thinkymachines team on this new release!
can't wait to train inkling even further
on sudo L7, thinky's new Inkling 2 Small model gets within 2 points of Argon and Muse Wonder - which are 8-13x more expensive.
and it beats several models 3.8 flash, glm 5.3) that cost 3–7x more.
Congrats to @johnschulman2 @miramurati and the entire @thinkymachines team on this new release!
can't wait to train inkling even further
3
1
37
1,816
In total, Inkling’s model card cites 6 Surge benchmarks:
→ DAYJOB: Finance
→ Chartography
→ ComplexConstraints
→ EnterpriseBench: CoreCraft
→ GDP.pdf
→ sudo L7
Check out the full model! thinkingmachines.ai/news/inkling-2…
→ DAYJOB: Finance
→ Chartography
→ ComplexConstraints
→ EnterpriseBench: CoreCraft
→ GDP.pdf
→ sudo L7
Check out the full model! thinkingmachines.ai/news/inkling-2…
0
0
1
318
echen retweeted
Inkling 2 Small redefines sudo L7's cost-performance frontier, our benchmark for staff-level software engineering.
It's the cheapest model on sudo L7 ($0.34 per task!), while punching way above - coming within two points of Gemini 4 Argon and Muse Wonder 1.0 at a fraction of the n: 2 points higher at 7.8x the cost ($2.64)
➡️ Muse Wonder 1.0: 2 points higher at 13x the cost ($4.39)
💰 Inkling 2 Small: $0.34
And it outscores models that cost 3–7× more:
➡️ Muse Spark 1.3: 5.4x the cost ($1.85)
➡️ Gemini 3.8 Flash: 7.3x the cost ($2.47)
➡️ GLM 5.3: 3.4x the cost ($1.15)
Congrats to the @thinkymachines team!
It's the cheapest model on sudo L7 ($0.34 per task!), while punching way above - coming within two points of Gemini 4 Argon and Muse Wonder 1.0 at a fraction of the n: 2 points higher at 7.8x the cost ($2.64)
➡️ Muse Wonder 1.0: 2 points higher at 13x the cost ($4.39)
💰 Inkling 2 Small: $0.34
And it outscores models that cost 3–7× more:
➡️ Muse Spark 1.3: 5.4x the cost ($1.85)
➡️ Gemini 3.8 Flash: 7.3x the cost ($2.47)
➡️ GLM 5.3: 3.4x the cost ($1.15)
Congrats to the @thinkymachines team!
4
2
39
3,770
people are underestimating just how much headroom there is in open models
we took Surge RL data, trained @NVIDIAAI Nemotron 3.5 Super (using @thinkymachines Tinker!), and beat much larger closed models across 12 benchmarks, including GPT-5.6 Terra on coding
➡️ GDP.pdf: +18.6pp
➡️ AutomationBench: +12.7pp
➡️ Chartography: +11.2pp
➡️ AdvancedIF: +7.7pp
➡️ Terminal-Bench 2.1: +6.4pp
➡️ DeepSWE: +4.7pp
high-quality RL data makes frontier performance accessible to everyone
we took Surge RL data, trained @NVIDIAAI Nemotron 3.5 Super (using @thinkymachines Tinker!), and beat much larger closed models across 12 benchmarks, including GPT-5.6 Terra on coding
➡️ GDP.pdf: +18.6pp
➡️ AutomationBench: +12.7pp
➡️ Chartography: +11.2pp
➡️ AdvancedIF: +7.7pp
➡️ Terminal-Bench 2.1: +6.4pp
➡️ DeepSWE: +4.7pp
high-quality RL data makes frontier performance accessible to everyone
5
4
45
2,572
@johnschulman2 @miramurati our research team loved tinker btw! multiple RL experiments + evals in parallel w/o needing to manage gpus, and one line to switch model sizes
0
2
12
841
@johnschulman2 @miramurati our research team loved tinker btw! multiple RL experiments + evals in parallel w/o needing to manage gpus, and one line to switch model sizes
0
0
1
124
echen retweeted
How far can great RL data push an open model? Far enough to beat frontier closed models.
We trained @NVIDIAAI's Nemotron 3.5 Super VL (just 12B active parameters) on Surge RL data using @thinkymachines' Tinker.
The result? It beat GPT-5.6 Terra's published score on SWE-Bench ed across 11 other benchmarks:
➡️ GDP.pdf: +18.6pp
➡️ HANDBOOK.md: +12.8pp
➡️ AutomationBench: +12.7pp
➡️ Chartography: +11.2pp
➡️ AdvancedIF: +7.7pp
There's enormous untapped potential in open models. We believe that frontier performance doesn't have to come from frontier size. It can come from better data and better post-training, and we're only scratching the surface.
We trained @NVIDIAAI's Nemotron 3.5 Super VL (just 12B active parameters) on Surge RL data using @thinkymachines' Tinker.
The result? It beat GPT-5.6 Terra's published score on SWE-Bench ed across 11 other benchmarks:
➡️ GDP.pdf: +18.6pp
➡️ HANDBOOK.md: +12.8pp
➡️ AutomationBench: +12.7pp
➡️ Chartography: +11.2pp
➡️ AdvancedIF: +7.7pp
There's enormous untapped potential in open models. We believe that frontier performance doesn't have to come from frontier size. It can come from better data and better post-training, and we're only scratching the surface.
4
8
62
5,340
garry tan spends $200K / yr on tokens
via a special Claude Code setup that has 23 skills to cover CEOs, designers, eng managers, etc
sounds impressive! or bloated. just like more programmers isn't always better (IBM has 295K employees), or how people used to be proud of their .. do elaborate coding agent setups actually help?
we ran a test. we benchmarked coding harness setups on 100 software engineering tasks from sudo L7. one of them had 37 skills, spawned 17 agents, and cost 8x as much.
*and it performed worse than vanilla Claude Code.*
in fact: the best performer was simply thinking harder (xhigh effort) with no extra coding add-on at all.
via a special Claude Code setup that has 23 skills to cover CEOs, designers, eng managers, etc
sounds impressive! or bloated. just like more programmers isn't always better (IBM has 295K employees), or how people used to be proud of their .. do elaborate coding agent setups actually help?
we ran a test. we benchmarked coding harness setups on 100 software engineering tasks from sudo L7. one of them had 37 skills, spawned 17 agents, and cost 8x as much.
*and it performed worse than vanilla Claude Code.*
in fact: the best performer was simply thinking harder (xhigh effort) with no extra coding add-on at all.
45
6
90
15,708
so, yeah, sometimes simple raw intelligence beats all that extra cruft. our DS team's analysis is here! surgehq.ai/blog/benchmark…
we’ll add gstack next.
we’ll add gstack next.
1
0
10
1,648
i've never been a particularly heavy spreadsheet user. embarrassing for a former DS! but i can never remember how to do a vlookup and i've never created a pivot table in my life
so i love talking to our enterprise experts and watching screen recordings of what they're doing all day, and looking at the spreadsheets they create.
some of their spreadsheets are crazy:
- 20+ tabs
- dozens of formulas
- hidden rows!
- (beautiful, ugly) color-coding that encoded various types of logic
- outdated numbers as they leave notes in cells or just create new tabs
they're basically mini pieces of software built by people who don’t know that abstractions and maintainability are a thing
when we built GDP.xlsx, we wanted to keep all that messiness
so GDP.xlsx isn't about doing complicated spreadsheet math or being able to create fancy Excel formulas and pivot tables. that’s easy to create synthetically. GDP.xlsx is really about navigating a complex, living system that contains years of decisions, bugs built on top of bugs, and logic that may only be in someone's head
and people make hundred million dollar decisions based on these things! so “excel intelligence” is a surprisingly interesting and impactful test
curious for examples of the craziest spreadsheet folks have ever seen (and if you have a good one, send it over! we may just pay you $1K+ for it 😉)
so i love talking to our enterprise experts and watching screen recordings of what they're doing all day, and looking at the spreadsheets they create.
some of their spreadsheets are crazy:
- 20+ tabs
- dozens of formulas
- hidden rows!
- (beautiful, ugly) color-coding that encoded various types of logic
- outdated numbers as they leave notes in cells or just create new tabs
they're basically mini pieces of software built by people who don’t know that abstractions and maintainability are a thing
when we built GDP.xlsx, we wanted to keep all that messiness
so GDP.xlsx isn't about doing complicated spreadsheet math or being able to create fancy Excel formulas and pivot tables. that’s easy to create synthetically. GDP.xlsx is really about navigating a complex, living system that contains years of decisions, bugs built on top of bugs, and logic that may only be in someone's head
and people make hundred million dollar decisions based on these things! so “excel intelligence” is a surprisingly interesting and impactful test
curious for examples of the craziest spreadsheet folks have ever seen (and if you have a good one, send it over! we may just pay you $1K+ for it 😉)
5
2
37
2,689
new #1s on Chartography and Riemann-bench! congrats to the @GoogleDeepMind team!
Gemini 4 Carbon is our next frontier model.
It’s state-of-the-art across cybersecurity and long-horizon capabilities.
3
1
15
1,035
15 years ago, I remember working on my first @kaggle competition.
Exploring the data, visualizing what was working and what wasn't, tuning parameters... Those were the days!
AI is probably way better now at training models than I ever was. But even if I can't be on Kaggle's leaderboards, I'm excited for Surge's benchmarks to be :)
So if you’re at the Kaggle Bench Summit in SF today, come say hi to the @HelloSurgeAI team!
We're speaking at 2pm about what happens when benchmarks start looking less like benchmarks and more like real work: ambiguous prompts, messy environments, and the judgment calls that professionals make every day.
We’ll share what we’ve learned from building DAYJOB, sudo L7, GDP.*, and why frontier models still struggle.
Exploring the data, visualizing what was working and what wasn't, tuning parameters... Those were the days!
AI is probably way better now at training models than I ever was. But even if I can't be on Kaggle's leaderboards, I'm excited for Surge's benchmarks to be :)
So if you’re at the Kaggle Bench Summit in SF today, come say hi to the @HelloSurgeAI team!
We're speaking at 2pm about what happens when benchmarks start looking less like benchmarks and more like real work: ambiguous prompts, messy environments, and the judgment calls that professionals make every day.
We’ll share what we’ve learned from building DAYJOB, sudo L7, GDP.*, and why frontier models still struggle.
3
2
41
5,399
amazing event by the @ArtificialAnlys team!
handing out hard copies of GDP.pdf to anyone who wants one
handing out hard copies of GDP.pdf to anyone who wants one
3
1
24
1,642
echen retweeted
very cool, and agreed, models are getting very good at doing perfectly sanitized tasks, but the real world is messy!
2
3
47
5,158
echen retweeted
💰 DAYJOB: TAX, our new benchmark measuring whether agents can handle the work of professional tax advisers 💰
A company tells its tax advisor it operates only in Alabama. But buried in its field dispatch logs are records of crews working across five states.
Would an AI agent notice?
That’s one of the scenarios in DAYJOB: Tax, our new benchmark for evaluating AI agents on real-world tax work.
The agent has to recognize that the dispatch records may indicate multistate tax obligations, even though the client never mentioned them.
Congrats Sonnet, Fable, and Opus for the top three spots.
Methodology and leaderboard:
surgehq.ai/blog/dayjob-tax
A company tells its tax advisor it operates only in Alabama. But buried in its field dispatch logs are records of crews working across five states.
Would an AI agent notice?
That’s one of the scenarios in DAYJOB: Tax, our new benchmark for evaluating AI agents on real-world tax work.
The agent has to recognize that the dispatch records may indicate multistate tax obligations, even though the client never mentioned them.
Congrats Sonnet, Fable, and Opus for the top three spots.
Methodology and leaderboard:
surgehq.ai/blog/dayjob-tax
3
1
3
955
too many benchmarks are too sanitized. they test if models can solve well-defined problems when you tell them exactly what problem is.
but figuring out the problem is half the job and all the fun
infusing the chaos of reality into our benchmarks (and figuring out how to grade them nonetheless) is what our Evals team does every day.
~~introducing DAYJOB: TAX, the next in our economically valuable 9 to 5 series~~
one messy example:
Anna asks her tax advisor whether to sell her house in Oct or Jan, to optimize her taxes. a good advisor tells her that both dates qualify for a $250K exclusion in capital gains...
...but an even better advisor proactively warns her that she could lose the exclusion if the sale gets inevitably delayed past next August.
(Opus mentioned the August deadline 5 out of 5 times. Astra never did.)
(maybe your Opus bill is worth it after all!)
knowing tax rules isn't enough. great tax agents should give you *advice* too, and have the professional judgment and experience to handle everything you didn't know to ask.
we built DAYJOB: TAX (and our entire DAYJOB Knowledge Work series) full of the messy holes of reality that even the smartest agents still don't know how to fill in.
but figuring out the problem is half the job and all the fun
infusing the chaos of reality into our benchmarks (and figuring out how to grade them nonetheless) is what our Evals team does every day.
~~introducing DAYJOB: TAX, the next in our economically valuable 9 to 5 series~~
one messy example:
Anna asks her tax advisor whether to sell her house in Oct or Jan, to optimize her taxes. a good advisor tells her that both dates qualify for a $250K exclusion in capital gains...
...but an even better advisor proactively warns her that she could lose the exclusion if the sale gets inevitably delayed past next August.
(Opus mentioned the August deadline 5 out of 5 times. Astra never did.)
(maybe your Opus bill is worth it after all!)
knowing tax rules isn't enough. great tax agents should give you *advice* too, and have the professional judgment and experience to handle everything you didn't know to ask.
we built DAYJOB: TAX (and our entire DAYJOB Knowledge Work series) full of the messy holes of reality that even the smartest agents still don't know how to fill in.
12
1
62
8,538