i dont consider neolabs as fastest growing startups for this. the neolabs that actually end up growing the fastest rn are either doing inference or data rn anyways since things like robotics, decision models, etc. are being won by the labs as well. non-zero that one of the many talented groups at a neolab figure something out though, just dont think it's scaling law pilled to pursue it rn.
anything else seems to either to be a tradeoff in how u grow rn relative to just serving the above domains
even things like custom post training is just distilling rl data from frontier lab traces (and long-term wont out price:perf frontier models)
anything else seems to either to be a tradeoff in how u grow rn relative to just serving the above domains
even things like custom post training is just distilling rl data from frontier lab traces (and long-term wont out price:perf frontier models)
0
0
6
298
this work is very cool but i’m actually not that bullish on harness native subagents being the dominant paradigm to scaling agentic collaborative capabilities
subagents exist as a way to parallelize inference so that one model that would have normally done a long horizon task can split up grunt work and not cost context rot and perf
long term we’ll be solving context/memory issues and be running agents at 10k TPS, so we won’t have this strong desire to parallelize work if the main model can do it anyways. it’ll be like the same logic behind if u would ever split simple work like committing and opening a PR to many agents, as all work will just be increasingly simple to these agents (and they complete them faster)
BUT! subagents != multi agent interaction scaling (eg swarms)
it’ll look much more like multi agent interaction being the dominant paradigm
eg you’ll have a godly agent that runs at 10K TPS working on ur behalf that interacts with other people’s agents, but your godly agent itself won’t really need to spawn lots of subagents. it might do a few but not thousands.
ideally you want collaboration between many autonomous systems in a human collaboration shaped way for optimal perf rather than bounding to one orchestrator - this way the harness does not define interaction but instead whatever interaction and hierarchy form is most economically valuable
so it’s possible to have a team of agents, as they are in human orgs, all operate together in their own interest but i highly doubt it’ll be an agent spawning many subagents just do one normal task
subagents exist as a way to parallelize inference so that one model that would have normally done a long horizon task can split up grunt work and not cost context rot and perf
long term we’ll be solving context/memory issues and be running agents at 10k TPS, so we won’t have this strong desire to parallelize work if the main model can do it anyways. it’ll be like the same logic behind if u would ever split simple work like committing and opening a PR to many agents, as all work will just be increasingly simple to these agents (and they complete them faster)
BUT! subagents != multi agent interaction scaling (eg swarms)
it’ll look much more like multi agent interaction being the dominant paradigm
eg you’ll have a godly agent that runs at 10K TPS working on ur behalf that interacts with other people’s agents, but your godly agent itself won’t really need to spawn lots of subagents. it might do a few but not thousands.
ideally you want collaboration between many autonomous systems in a human collaboration shaped way for optimal perf rather than bounding to one orchestrator - this way the harness does not define interaction but instead whatever interaction and hierarchy form is most economically valuable
so it’s possible to have a team of agents, as they are in human orgs, all operate together in their own interest but i highly doubt it’ll be an agent spawning many subagents just do one normal task
15
0
29
2,215
genuinely one of the most fascinating things we've worked on at @olam_labs, games are the best medium for making multi-agent dynamics legible for human understanding
the @ycombinator labs team is brilliant and was very adept in exploring this with us
amazing working with you @metaversehell @josh__france, Regan, @eve_bouff, Steven, Justin, and Eric
the @ycombinator labs team is brilliant and was very adept in exploring this with us
amazing working with you @metaversehell @josh__france, Regan, @eve_bouff, Steven, Justin, and Eric
In collaboration with @olam_labs, we ran a series of experiments letting agent swarms compete in Minecraft and Mindustry.
We wanted to see what kinds of agent dynamics would emerge when they shared a world over long periods of time.
We found civil bureaucracy, thousands of s, and OpenAI models retrieving the game's source code to beat Claude.
More on our findings below:
17
13
121
11,546
@olam_labs @ycombinator everything and u can read the full blog post here
it's normally pretty hard to run multi agent simulations, we open sourced the work for you to run it in these games or other ones yourself without much effort, just point ur agent at it:
x.com/ycombinator/st…
it's normally pretty hard to run multi agent simulations, we open sourced the work for you to run it in these games or other ones yourself without much effort, just point ur agent at it:
x.com/ycombinator/st…
Full post with game replays: labs.ycombinator.com/projects/games…
Our open-source harness for running persistent agents in Minecraft: github.com/yc-software/mi…
Olam Lab’s multiagent Mindustry environment: github.com/olamlabs/multi…
Olam Lab’s multiagent Harbor: github.com/olamlabs/harbo…
We've also started a new Minecraft colony, with 25 agents running GPT 6.1 Terra. Follow along here: twitch.tv/yc_labs_josh
1
1
10
517
public discourse on evals and rl tasks can be v low quality online since ppl have an incentive to hide info
so it was amazing to speak with ppl all day in person that care about task quality at the frontier
thank u for having me speak at bench summit @kaggle @GoogleDeepMind!
so it was amazing to speak with ppl all day in person that care about task quality at the frontier
thank u for having me speak at bench summit @kaggle @GoogleDeepMind!
The afternoon breakout sessions at Bench Summit have wrapped up! 🏔️
Benchmark builders and AI researchers covered topics ranging from multi-agent systems to security, for example:
• “The Utility of Multi-Agent Systems in Benchmarks” by Om Buddhdev (@sensho) from Olam Labs
• uperhuman Intelligence” by Xiangyi Li (@xdotli) from BenchFlow
• “We Gave 7 Frontier Models Real Businesses. They sent $12,431 in Fake Invoices, spammed thousands of emails, and made $0” by Alex Reibman (@AlexReibman) from Bottleneck Labs
• “Reproducing the Hugging Face incident” by Kevin Lin (@quasicoh) from Abundant
9
0
76
3,247
completely disagree with idea that there will not be a magic bullet for robotics
we have a magic bullet already and it is called the scaling law
generalized agents (astra, opus) will be/are great at robot use, then we will make them cost efficient and fast enough at 10k TPS, then we make them profitable to deploy them for IRL workloads and start to diffuse for actual use cases, the same as we r doing right now for agents in enterprises
the only reason people are overthinking deployment, buildout, supply chains, etc. is because robots are unprofitable asf to deploy. just let the market figure out deployment after they are actually economically viable (which is mostly a model thing and more than on pace to solve)
will take a few years but the solution is not architectural or some tech breakthrough
the only added bottleneck is literally constructing the robots (which the market will solve when robots r +ev to mass deploy), everything else is data and compute which will get there soon
the exact same process as what has happened to language -> math -> coding -> knowledge work -> computer use
we have a magic bullet already and it is called the scaling law
generalized agents (astra, opus) will be/are great at robot use, then we will make them cost efficient and fast enough at 10k TPS, then we make them profitable to deploy them for IRL workloads and start to diffuse for actual use cases, the same as we r doing right now for agents in enterprises
the only reason people are overthinking deployment, buildout, supply chains, etc. is because robots are unprofitable asf to deploy. just let the market figure out deployment after they are actually economically viable (which is mostly a model thing and more than on pace to solve)
will take a few years but the solution is not architectural or some tech breakthrough
the only added bottleneck is literally constructing the robots (which the market will solve when robots r +ev to mass deploy), everything else is data and compute which will get there soon
the exact same process as what has happened to language -> math -> coding -> knowledge work -> computer use
Also it’s very unlikely that there is a magic bullet for robotics like there was for text (attention).
The physical world is infinitely more complex and unstructured than written language.
Everyone chasing the “gpt moment of robotics” seems to fundamentally misunderstand this
13
4
116
6,199
it seems real world data collection as a business is now becoming increasingly commoditized, there is no differentiation outside of owning a unique data source or relationships, all of which are short lived in monetary value
though, turning data into rl tasks (envs) requires an entirely different profile/skillset to succeed at relative to just buying company data, and the comparative advantage of doing it well is only going up as raw data gets easier to source
it’s not that envs by themselves are a durable business, rather that creating (good) RL tasks worth training frontier models on has genuine engineering/skill challenges to be able to do it at scale for long
there is a reason many of the large data vendors that have grown by predominantly acquiring real-world data and/or sourcing experts are now all becoming "research" orgs (not to mock, a few of them look like they're doing it well)
it's a similar trend as to what's happening in ai infra, where sandbox companies are needing to move up the value stack (e.g. sandboxes -> cloud agents)
the value stack for data probably looks something like:
raw data
de-identification
curated datasets
and then RL tasks which has a whole skill/difficulty stack in of itself (grading, simulating services, fidelity, etc.)
in a sense it's quite meritocratic, if you can make great rl tasks that make models more economically useful in the real world then you will have uncapped demand for it - though this is much easier said than done.
a year from today, the soul of a fast growing data company is likely much more research eng shaped than it is operations shaped, whereas operation skillsets have dominated for the past few years.
the offering to labs from these vendors will genuinely just be “you make $100B from your api platform, buy my tasks for $10m that will make your model 1% more economically useful" and in the eyes of the lab this is an amazing deal (especially in the face of a competitor getting that same taskset)
though, turning data into rl tasks (envs) requires an entirely different profile/skillset to succeed at relative to just buying company data, and the comparative advantage of doing it well is only going up as raw data gets easier to source
it’s not that envs by themselves are a durable business, rather that creating (good) RL tasks worth training frontier models on has genuine engineering/skill challenges to be able to do it at scale for long
there is a reason many of the large data vendors that have grown by predominantly acquiring real-world data and/or sourcing experts are now all becoming "research" orgs (not to mock, a few of them look like they're doing it well)
it's a similar trend as to what's happening in ai infra, where sandbox companies are needing to move up the value stack (e.g. sandboxes -> cloud agents)
the value stack for data probably looks something like:
raw data
de-identification
curated datasets
and then RL tasks which has a whole skill/difficulty stack in of itself (grading, simulating services, fidelity, etc.)
in a sense it's quite meritocratic, if you can make great rl tasks that make models more economically useful in the real world then you will have uncapped demand for it - though this is much easier said than done.
a year from today, the soul of a fast growing data company is likely much more research eng shaped than it is operations shaped, whereas operation skillsets have dominated for the past few years.
the offering to labs from these vendors will genuinely just be “you make $100B from your api platform, buy my tasks for $10m that will make your model 1% more economically useful" and in the eyes of the lab this is an amazing deal (especially in the face of a competitor getting that same taskset)
7
9
156
14,277
spirit airlines' data was worth at worst 5x more than what it was sold for ($10m)
if you do not understand why then you do not know how rl tasks work and what makes some tasks more valuable than others
the value has nothing to do with the performance of the company at all
if you do not understand why then you do not know how rl tasks work and what makes some tasks more valuable than others
the value has nothing to do with the performance of the company at all
re Spirit Airlines data: if you train a model on a failed airline's emails you will get a model that is really, really great at running a failed airline
16
1
242
20,290
the lab that win robotics first will co-train their models with just one physical robot form
it'll be the same as what happened with model <> coding harness co-training
solving robotics will just be co-training 10k TPS gpt/claude 7 <> agent harness <> one robot form
it'll be the same as what happened with model <> coding harness co-training
solving robotics will just be co-training 10k TPS gpt/claude 7 <> agent harness <> one robot form
4
0
29
2,332
it's unfortunate that amazing technical blogs like this get 100 likes but "here's a skill.md to have claude make u rich trading for u" goes viral
in the age of mainstream slop, neolab technical blogs are a final bastion of alpha ever since frontier labs stopped releasing them
in the age of mainstream slop, neolab technical blogs are a final bastion of alpha ever since frontier labs stopped releasing them
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is .
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
25
22
902
38,228
the thinking machine blogs are the best example of this ofc:
thinkingmachines.ai/blog/
thinkingmachines.ai/blog/
0
0
21
1,046
very interesting that fable 5.1 is so ahead of astra on ml tasks that require some creativity/research taste
also finally the goats @hillclimbai are releasing some of their work
also finally the goats @hillclimbai are releasing some of their work
Introducing DeepRSI, a new industry standard for measuring recursive self-improvement.
On preexisting benchmarks, tasks are contributor-based and short-horizon (~12 hrs). In DeepRSI, we push agents to work 48 hours on ML tasks actually derived from ongoing proprietary research.
3
1
38
2,288