Fascinating talk from @criticalneuro on neuroscience-inspired generative AI at the SSI-FM workshop #ICLR2025 🧠🤖
1
5
47
With a stellar lineup of speakers and panelists, including Yoshua Bengio 🙀, the Scaling Self-Improving Foundation Models at @iclr_conf promises to be 🔥
⏰ Sunday, April 27
📍 Garnet 214-215
⏰ Sunday, April 27
📍 Garnet 214-215
2
21
145
I’ll be in Singapore 🇸🇬 for #ICLR this week. Keen to connect with old friends and meet new folks interested in open-endedness, LLM agents, and accelerating scientific discovery with AI.
I’ll be giving a talk on “Automating Scientific Discovery: How Far Are We?” at the fantastic ay and co-organizing the Self-Improving Foundation Models workshop on Sunday. Finally, don’t miss out @MartinKlissarov’s oral on MaestroMotif.
I’ll be giving a talk on “Automating Scientific Discovery: How Far Are We?” at the fantastic ay and co-organizing the Self-Improving Foundation Models workshop on Sunday. Finally, don’t miss out @MartinKlissarov’s oral on MaestroMotif.
3
12
98
We also introduce a hierarchical framework for evaluating the capabilities of advanced AI Research Agents.
1
0
15
Our analysis highlights that the agents take a structured approach for solving AI Research tasks, where they begin with getting familiar with the environment and the task, and then they conduct multiple iterations of inspecting the code, editing it, training models, evaluating e with a submission.
Check out the paper for more analysis of the agents’ behaviors on MLGym.
Check out the paper for more analysis of the agents’ behaviors on MLGym.
1
0
14
While OpenAI O1-Preview is the best-performing model, it is also the most computationally expensive by a wide margin. In contrast, Gemini-1.5-Pro and Claude-3.5- Sonnet are much more cost-effective while still reaching high performance not too far from OpenAI O1’s.
the best balance between performance and cost on MLGym-Bench, being the cheapest model to run (9x cheaper than O1) while achieving 99% of O1’s AUP.
While relatively cheap to run, GPT-4o and Llama-3.1-405b perform worse than the other models.
the best balance between performance and cost on MLGym-Bench, being the cheapest model to run (9x cheaper than O1) while achieving 99% of O1’s AUP.
While relatively cheap to run, GPT-4o and Llama-3.1-405b perform worse than the other models.
1
0
12
We propose to use performance profile curves and AUP scores to compare the performance of multiple agents on MLGym tasks. This allows us to compare the relative performance gains of a method (agent-model pair) over a set of diverse tasks. We measure the agent’s best attempt and 4 trials.
We find that OpenAI O1-preview is the best-performing model on aggregate across our set of tasks for both Best Attempt and Best Submission, with Gemini-1.5-Pro and Claude-3.5-Sonnet being close behind.
We find that OpenAI O1-preview is the best-performing model on aggregate across our set of tasks for both Best Attempt and Best Submission, with Gemini-1.5-Pro and Claude-3.5-Sonnet being close behind.
1
0
12
MLGym-Bench contains a set of diverse open-ended AI research tasks from a wide-range of domains such as CV, NLP, RL, and game theory.
1
0
14
The memory module is crucial so the agent can manage an experimental log and draw useful insights from it for long-horizon research tasks. In particular, the memory tool enables the agent to retrieve the best configuration found so far and continue to iterate on it. Here, you can gent doing precisely this.
1
1
15
The MLGym environment comes with a number of tools defined as bash or python scripts, including a memory module and a literature search tool. The agent can also interact with the shell through any bash command.
1
1
15
MLGym is the first agentic framework for AI Research Agents based on a Gymnasium interface.
MLGym-bench is also the only benchmark that includes algorithmic tasks such as game theory and logic reasoning.
In contrast with many existing benchmarks for AI Research Agents, MLGym nded research tasks spanning a wide range of domains. It also comes with an agentic harness and allows for flexible evaluation artifacts such as model checkpoints, RL algorithms, or code representing game theoretical strategies.
MLGym-bench is also the only benchmark that includes algorithmic tasks such as game theory and logic reasoning.
In contrast with many existing benchmarks for AI Research Agents, MLGym nded research tasks spanning a wide range of domains. It also comes with an agentic harness and allows for flexible evaluation artifacts such as model checkpoints, RL algorithms, or code representing game theoretical strategies.
1
0
16
MLGym is a unified framework for evaluating and developing LLM agents for solving AI Research tasks. MLGym is a Gymnasium environment that can execute shell commands in a local docker-machine shell. It provides access to four core components: Agents, Environment, Datasets, and sign allows one to utilize and extend the library easily. For example, researchers can easily integrate RL algorithms, implement other agentic harnesses to improve performance, they can expand the environment by adding more tools for an agent, add more datasets within a given task, and even add more tasks to the MLGym benchmark.
1
3
30
Super excited to share 🧠MLGym 🦾 – the first Gym environment for AI Research Agents 🤖🔬
We introduce MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks.
The key contributions of our work are:
🕹️ Enables the t training algorithms for AI Research Agents such as RL
🛠️ Provides a flexible evaluation framework that can accommodate different artifacts such as models, algorithms, or predictions
🤖 Allows researchers to evaluate any model without the need to develop a custom agentic harness
🎯 Introduces 13 diverse open-ended AI Research tasks for evaluating AI Research Agents on a wide range of domains such as computer vision, natural language processing, reinforcement learning, game theory, and logical reasoning.
📈 Proposes a new evaluation metric for AI Research Agents
MLGym makes it easy to:
1) Add new tasks
2) Evaluate new models
3) Integrate new agents
Check out a video of the MLGym Agent to see how it performs the full pipeline of idea generation💡, implementation 👩💻, experimentation 👩🔬, and iteration 🔄 to improve on ML tasks.
Huge thanks to the exceptionally talented @deepaknathani11 who led this work and to all the other amazing collaborators who made this possible 🙏🫶🚀
We introduce MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks.
The key contributions of our work are:
🕹️ Enables the t training algorithms for AI Research Agents such as RL
🛠️ Provides a flexible evaluation framework that can accommodate different artifacts such as models, algorithms, or predictions
🤖 Allows researchers to evaluate any model without the need to develop a custom agentic harness
🎯 Introduces 13 diverse open-ended AI Research tasks for evaluating AI Research Agents on a wide range of domains such as computer vision, natural language processing, reinforcement learning, game theory, and logical reasoning.
📈 Proposes a new evaluation metric for AI Research Agents
MLGym makes it easy to:
1) Add new tasks
2) Evaluate new models
3) Integrate new agents
Check out a video of the MLGym Agent to see how it performs the full pipeline of idea generation💡, implementation 👩💻, experimentation 👩🔬, and iteration 🔄 to improve on ML tasks.
Huge thanks to the exceptionally talented @deepaknathani11 who led this work and to all the other amazing collaborators who made this possible 🙏🫶🚀
15
120
482
The era of Hierarchical Agents has begun
Can AI agents adapt zero-shot, to complex multi-step language instructions in open-ended environments?
We present MaestroMotif, a method for AI-assisted skill design that produces highly capable and steerable hierarchical agents. To the best of our knowledge, it is the first pert labeled datasets, solves compositional tasks requiring hundreds of steps for completion.
All the modules within MaestroMotif are learned from interaction: from the highest level of planning to the lowest-level of sensorimotor control. On the open-ended domain of NetHack, it surpasses existing approaches, including those that are fine-tuned specifically for each task.
At the heart of MaestroMotif is the idea that decomposing a task into subtasks significantly helps decision making. MaestroMotif leverages an agent designer's intuition about a domain to identify important skills and describe them in natural language. These short descriptions then get converted into adaptable hierarchical agents through AI feedback and in-context learning.
Our paper was recently published at ICLR 2025 and we open-source the whole project including the code, prompts and pre-trained models.
Paper: arxiv.org/abs/2412.08542
Code: github.com/mklissa/maestr…
NotebookLM Podcast: bit.ly/4jLi6mo
This work was done with the amazing @HenaffMikael, @robertarail, @shagunsodhani, Pascal Vincent, @yayitsamyzhang, @pierrelux, Doina Precup, with equal supervision by @MarlosCMachado and @proceduralia.
Take a look at the following thread:
2
4
36
If you work on self-improving models, algorithms, systems, or applications, consider submitting your work to our ICLR workshop.
We have a fantastic lineup of speakers and panelists!
We have a fantastic lineup of speakers and panelists!
🚨 We are organizing an ICLR workshop on self-improving foundation models w/o human supervision at ICLR 2025 in Singapore!
Deadline: Feb 7, AoE (submit your ICML papers!)
Details: sites.google.com/berkeley.edu/s…
We have an amazing line up of speakers + panelists, more info coming soon.
1
10
80
Rainbow Teaming poster getting crowded. Come hear @sharathraparthy and @_andreilupu explaining how we can use open-ended learning methods like quality-diversity to generate endless adversarial prompts for LLMs and make them more robust in the process 🌈🌈🌈
@_andreilupu @robertarail and I are presenting Rainbow Teaming 🌈 at Set LLM workshop. Location - Schubert 5. Come chat with us!
2
5
30
@bogdan_mazoure kicking off the #genai4dm workshop #ICLR2024 highlighting the amazing progress in generative AI in the last decade
0
0
3
@proceduralia giving a talk on how you can use transformers to learn action world models to improve policy gradient and credit assignment in RL at the #genai4dm workshop at #ICLR2024
0
2
13
Inspiring talk from @katjahofmann on how we can empower game creators with learned action and world models! The video game generation results look very impressive! At the GenAI for Decision Making workshop #ICLR2024
0
2
31
I’ll be at #ICLR2024 on Saturday for the LLM Agents Workshop Panel 🚀
Some of my collaborators will also be there throughout the week presenting our work on:
- how RLHF affects LLM generalisation and diversity with @_robertkirk
- training NetHack agents using LLM feedback with duralia
- the generalization gap in offline RL with @ishitamed
Don’t miss our workshop on Generative Models for Decision Making on Friday which has a really great lineup of speakers 🤩 with @bogdan_mazoure & co.
Some of my collaborators will also be there throughout the week presenting our work on:
- how RLHF affects LLM generalisation and diversity with @_robertkirk
- training NetHack agents using LLM feedback with duralia
- the generalization gap in offline RL with @ishitamed
Don’t miss our workshop on Generative Models for Decision Making on Friday which has a really great lineup of speakers 🤩 with @bogdan_mazoure & co.
3
10
69
11) Giving a talk at GenPlan's workshop on "In-Context Learning of Sequential Decision-Making Tasks".
1
1
5
8) "A Study on Improving Reasoning in Language Models" led by @d_yuqing at the ICBINB workshop.
1
0
7
7) "A Study of Generalization in Offline Reinforcement Learning" led by @ishitamed and @qingfeiyou at the GenPlan workshop.
1
1
7
6) "Learning to Solve New Sequential Decision-Making Tasks with In-Context Learning" led by @sharathraparthy. Poster at FMDM workshop.
arxiv.org/abs/2312.03801
arxiv.org/abs/2312.03801
1
1
8
5) "Understanding the Effects of RLHF on LLM Generalisation and Diversity" led by @_robertkirk. Oral presentation at the Instruction Tuning and Instruction Following workshop.
arxiv.org/abs/2310.06452
arxiv.org/abs/2310.06452
1
0
7
4) "Motif: Intrinsic Motivation from Artificial Intelligence Feedback" led by @MartinKlissarov and @proceduralia. Oral at both FMDM and ALOE workshops.
arxiv.org/abs/2310.00166
arxiv.org/abs/2310.00166
1
0
8
3) "Improving Language Plasticity via Pretraining with Active Forgetting" led by @yihong_thu
arxiv.org/abs/2307.01163
neurips.cc/virtual/2023/p…
arxiv.org/abs/2307.01163
neurips.cc/virtual/2023/p…
1
0
7
2) "On the Importance of Exploration for Generalization in Reinforcement Learning" led by @yidingjiang
arxiv.org/abs/2306.05483
neurips.cc/virtual/2023/p…
arxiv.org/abs/2306.05483
neurips.cc/virtual/2023/p…
1
0
10
1) Toolformer's poster. Don't miss @JaneDwivedi's oral presentation.
arxiv.org/abs/2302.04761
neurips.cc/virtual/2023/p…
arxiv.org/abs/2302.04761
neurips.cc/virtual/2023/p…
1
0
10
CSP outperforms a number of popular
baselines on a wide range of scenarios from two challenging domains, Brax (locomotion) and Continual World (manipulation).
baselines on a wide range of scenarios from two challenging domains, Brax (locomotion) and Continual World (manipulation).
1
0
4
The learned subspace is diverse and compositional. Our method does not suffer from forgetting
and displays positive transfer to many new tasks, including combinations of previously seen ones.
and displays positive transfer to many new tasks, including combinations of previously seen ones.
1
0
3
The subspace’s high
expressivity allows CSP to perform well for many different tasks while growing
sublinearly with the number of tasks.
expressivity allows CSP to perform well for many different tasks while growing
sublinearly with the number of tasks.
1
0
3
We introduce Continual Subspace of
Policies (CSP), a new approach that incrementally builds a subspace of policies for
training a reinforcement learning agent on a sequence of tasks.
Policies (CSP), a new approach that incrementally builds a subspace of policies for
training a reinforcement learning agent on a sequence of tasks.
1
0
4
In this work, we aim to strike a better
balance between an agent’s size and performance by designing a method that grows
adaptively depending on the task sequence.
balance between an agent’s size and performance by designing a method that grows
adaptively depending on the task sequence.
1
0
2
PPO's value predictions have a quasi-linear dependency on the episode step, suggesting that it memorizes instance-specific features. In contrast, our approach DAAC, does not show the same trend in the advantage, suggesting it is less prone to overfitting.
1
0
1
We find a positive correlation between value loss and generalization when using a shared network for the policy and value, suggesting that there is a trade-off between fitting the value function and learning a policy that generalizes to new environments.
1
0
1
Our approach achieves a new SOTA on the Procgen benchmark and competitive results on a few DeepMind Control tasks with distractors.
1
0
1
We propose to use separate networks to learn the policy and value function. As an auxiliary task for training the policy network, we predict the advantage instead of the value. We find that this provides useful gradients while alleviating overfitting.
1
0
3
Many deep RL methods use a shared network to represent the policy and value function. However, more information is generally needed to accurately estimate the value than to learn the optimal policy, leading to poor generalization to new task instances.
1
0
4
Our work on “Decoupling Value and Policy for Generalization in Reinforcement Learning” w/ @rob_fergus will be presented as a long talk @icmlconf 2021. Come check it out in Poster Session 1 (Tuesday, 12 - 2pm ET / 5 - 7pm BST): icml.cc/virtual/2021/o….
1
15
64
The dynamics embeddings uncover the manifold structure of the true dynamics distribution, while the policy embeddings capture the identity of the policy used to generate them.
1
0
5
Our method outperforms other meta-learning approaches when evaluated on single episodes of new environments where fast adaptation is crucial.
1
0
4
