You can now run FrontierMemory to measure how well your model or harness improves with experience.
We’re also now accepting submissions to the official leaderboard.
Code and submission instructions are available at:
github.com/letta-ai/front…
We’re also now accepting submissions to the official leaderboard.
Code and submission instructions are available at:
github.com/letta-ai/front…
4
14
61
3,473
Honored to have been a featured ecosystem partner by @JensenHuang at GTC Berlin!
We're excited to partner with @nvidia on research towards developing agents that continually learn and self-improve.
We're excited to partner with @nvidia on research towards developing agents that continually learn and self-improve.
5
5
32
2,516
@vraj_ai Full repo to run the benchmark will be live shortly (~1 day or so, will be linked on the frontiermemory.com site), so you'll be to benchmark your own harnesses too.
1
0
2
86
@ALL_Jonah @MatthewBerman @OpenAI @AnthropicAI @claudeai Use Opus to orchestrate Codex subagents, creating and passing the relevant memories through to the next layer.
0
0
1
33
@dialectforge @charlespacker @MatthewBerman @OpenAI @AnthropicAI @claudeai You can run it locally! Full instructions on how to do so (with accompanying GitHub repo) will be live shortly (next ~day). Once live, the main website will link to it under the "Submit" tab.
0
0
0
39
Letta retweeted
Recommended reading and a super-interesting benchmark for measuring continual learning capabilities through memory.
Agent memory is largely unsolved. As agents advance in capabilities like long-horizon work, it's unclear which models or harnesses work best, or which memory e.
Opus 5.5 at the top is not a surprise. But it's exciting that open-weight models like GLM-5.3 perform so well on memory generation.
My big question is how this (better memory systems and persistent agent capabilities) translates into more performant, efficient RSI agents and harnesses.
Agent memory is largely unsolved. As agents advance in capabilities like long-horizon work, it's unclear which models or harnesses work best, or which memory e.
Opus 5.5 at the top is not a surprise. But it's exciting that open-weight models like GLM-5.3 perform so well on memory generation.
My big question is how this (better memory systems and persistent agent capabilities) translates into more performant, efficient RSI agents and harnesses.
Introducing FrontierMemory: a new way to evaluate agent memory through downstream task performance.
While existing memory benchmarks focus on recall (saturated via agentic search), FrontierMemory focuses on how agents learn from experience.
42
12
108
13,626
Letta retweeted
Memory is becoming a core part of agent harnesses/products (e.g. Instinct, @Muse, @cognition Devin, @PrimeIntellect Prime Agent, @NousResearch Hermes, @openclaw, @LangChain Deep Agents, @Letta_AI) -- yet despite every agent claiming to "continually learn", there's no benchmark to actually measure that capability.
A big problem is that memory benchmarks today only measure recall (e.g. simple Q/A about past events). They don't measure if/how agents actually learn from experience to get better at work.
At the same time, benchmarks for coding/knowledge work consist of many isolated, unrelated tasks - so they also don't allow for measuring cross-task learning. The context needed for a task is fully specified up-front (very unrealistic).
In practice, both agents and humans work through sequences of many *related* tasks. Experience from past work provides context and learnings that apply to future work. This is why if you're a manager, you have to provide a lot more context when assigning tasks to a brand new employee or external contractor. Meanwhile, a great senior employee might not need any instructions at all.
With FrontierMemory, we adapt existing benchmark tasks for coding and knowledge work so that instead of all the necessary context being provided up-front and within the task, it has to be learned from past experience. This allows us to compare how well different models and harnesses learn over time.
A big problem is that memory benchmarks today only measure recall (e.g. simple Q/A about past events). They don't measure if/how agents actually learn from experience to get better at work.
At the same time, benchmarks for coding/knowledge work consist of many isolated, unrelated tasks - so they also don't allow for measuring cross-task learning. The context needed for a task is fully specified up-front (very unrealistic).
In practice, both agents and humans work through sequences of many *related* tasks. Experience from past work provides context and learnings that apply to future work. This is why if you're a manager, you have to provide a lot more context when assigning tasks to a brand new employee or external contractor. Meanwhile, a great senior employee might not need any instructions at all.
With FrontierMemory, we adapt existing benchmark tasks for coding and knowledge work so that instead of all the necessary context being provided up-front and within the task, it has to be learned from past experience. This allows us to compare how well different models and harnesses learn over time.
Introducing FrontierMemory: a new way to evaluate agent memory through downstream task performance.
While existing memory benchmarks focus on recall (saturated via agentic search), FrontierMemory focuses on how agents learn from experience.
31
15
215
16,310
Letta retweeted
FrontierMemory from @Letta_AI is the first benchmark to measure memory based on performance on downstream tasks.
Memory remains one of the biggest unsolved challenges in AI.
FrontierMemory ACTUALLY gives us a way to actually measure and compare memory capabilities across different harnesses and models.
Memory remains one of the biggest unsolved challenges in AI.
FrontierMemory ACTUALLY gives us a way to actually measure and compare memory capabilities across different harnesses and models.
Introducing FrontierMemory: a new way to evaluate agent memory through downstream task performance.
While existing memory benchmarks focus on recall (saturated via agentic search), FrontierMemory focuses on how agents learn from experience.
2
3
15
1,181
Letta retweeted
Truly capable agents must learn from experience, not merely recall facts. The FrontierMemory benchmark evaluates agent memory directly by measuring its impact on downstream task performance.
Introducing FrontierMemory: a new way to evaluate agent memory through downstream task performance.
While existing memory benchmarks focus on recall (saturated via agentic search), FrontierMemory focuses on how agents learn from experience.
10
7
55
3,461
Thanks to our partners for providing early feedback and inference to support the benchmark @AnthropicAI @OpenAI @nvidia @databricks @Snowflake @baseten @FireworksAI_HQ
0
1
7
1,214
FrontierMemory is a continuous benchmark. In our next release, we plan to add longer-horizon tasks that enable RSI from memory.
For more details see: frontiermemory.com
For more details see: frontiermemory.com
1
1
20
1,582
FrontierMemory also allows measuring the ability of agent harnesses to improve over time. We find that Letta Dream and @PrimeIntellect's Prime Agent improve the most from learning from prior experience data.
6
2
15
4,860
Letta retweeted
Hey, if you're still using @Letta_AI Classic like I am, you need an ADE (as Letta has EOL'd and retired the online version).
I'm maintaining the Sanctum branch of both Letta and the ADE here:
github.com/sanctumos/letta
github.com/sanctumos/ade
I'm maintaining the Sanctum branch of both Letta and the ADE here:
github.com/sanctumos/letta
github.com/sanctumos/ade
0
3
8
498