Thank you, Rich! Most robot learning happens in simulation or on offline datasets, and stops once a policy is deployed. Long-term autonomy requires agents that keep learning from runtime experience, as animals do. It's been a privilege to work on these ideas with wonderful folks!
And here is the new PhD thesis of Gautham Vasan (@Gautham529), whose committee I was recently on. I thought this thesis was particularly well done, including the AVG algorithm in Chapter 6, which is a new streaming RL actor-critic (different from the Stream-X algorithms).
Title: Robots That Learn on the Fly Through Real-World Interaction
URL: gauthamvasan.com/papers/Vasan_G…
Abstract:
A robot that learns from its own runtime experience can improve its behavior over its operational lifetime and adapt to conditions that were never anticipated. The prevailing practice in robot learning provides no such ability: practitioners train a control policy before deployment, from human-provided data or from simulated interaction, and hold it fixed afterward. When performance degrades after deployment, a human engineer may need to diagnose the failure, collect more data or refine the simulator, and retrain and redeploy the policy. Reinforcement learning (RL) from ongoing interaction could reduce reliance on humans for data collection, retraining, and redeployment.
Real-time reinforcement learning names the setting in which an agent senses, acts, and learns from ongoing interaction as wall-clock time advances for both the agent and the environment. Real-time RL on physical robots imposes three practical demands: (a) learning must operate within the robot’s limited computational and memory resources; (b) learning must reach competent performance reliably within a practical wall-clock budget; (c) learning must use rewards that are computable from the robot’s own sensing and simple to specify across tasks. I present three complementary contributions that address the three demands.
The first contribution, the Remote-Local Distributed (ReLoD) framework, supplements the robot’s limited computational resources with a remote workstation. The framework distributes action computation and learning updates between the two computers. Experiments on vision-based tasks showed that the performance of an agent using Soft Actor-Critic, a widely used batch RL algorithm that learns from stored transitions, degraded under limited onboard computation. A robot using ReLoD recovered performance by drawing on remote computation. Across repeated runs, competent policies were learned from scratch within a few hours.
The second contribution is an empirical study of the minimum-time task specification, which assigns a reward of −1 at each timestep and terminates the episode when the goal is reached. Given matched interaction budgets, final policies learned under the minimum-time specification matched or outperformed those learned under guiding reward specifications, which reward progress toward the goal. Using the minimum-time specification, four physical robots learned goal-reaching policies from raw-pixel observations within hours and without external instrumentation.
The third contribution is Action Value Gradient (AVG), a computationally cheap streaming RL algorithm that learns with neural networks. An agent using AVG updates its parameters once from each transition (an observation, action, reward, and next observation) and stores no transitions for reuse. On simulated tasks, agents using AVG outperformed agents using existing streaming RL methods, often achieving final performance comparable to batch RL methods. On physical-robot tasks, agents using AVG learned competent policies within hours on an edge device.
The three contributions address the three demands in complementary ways. ReLoD and AVG meet the computational resource demand from opposite directions: ReLoD supplements a robot’s onboard computer with remote computation when feasible, and AVG is computationally cheap, enabling on-device learning. The minimum-time specification reduces specifying a goal-reaching task to one requirement, detecting the goal from onboard sensing. Across the contributions, physical robots learned competent behavior from scratch within hours. Together, the contributions advance real-time reinforcement learning on physical robots toward the longer-term goal of continual adaptation and long-term autonomy.
Gautham is now a research scientist at Zyphra.
1
0
31
895
Gautham Vasan retweeted
Check out our latest work with Mohamed Elsayed (@mhmd_elsaye), Gautham Vasan (@Gautham529), and Rupam Mahmood (@rupammahmood).
arxiv.org/pdf/2410.14606
Stream-X doesn't just work across a wide range of simulated environments, but also on-board our Open Ant (@escaped_sim_ant) n-based heading task: using vision to orient itself toward a desired heading.
We ran the experiments in a lab at @NASAJPL, with researchers moving robots around and people walking through the environment, creating the kind of changing conditions that real robots have to deal with. But the robot didn't care, it kept learning and learning over many hours and days.
arxiv.org/pdf/2410.14606
Stream-X doesn't just work across a wide range of simulated environments, but also on-board our Open Ant (@escaped_sim_ant) n-based heading task: using vision to orient itself toward a desired heading.
We ran the experiments in a lab at @NASAJPL, with researchers moving robots around and people walking through the environment, creating the kind of changing conditions that real robots have to deal with. But the robot didn't care, it kept learning and learning over many hours and days.
0
3
14
524
Gautham Vasan retweeted
@AmiiThinks @MPI_IS If you are interested in working with me at *the* RL powerhouse @UAlberta on robot learning on physical robots, please drop me a message.
Retweets welcome 🙏
Retweets welcome 🙏
3
14
31
8,418
Congrats @RichardSSutton and Andy! You've built an incredible community and shaped the thinking of many researchers, including myself. Well deserved!
Meet the recipients of the 2024 ACM A.M. Turing Award, Andrew G. Barto and Richard S. Sutton! They are recognized for developing the conceptual and algorithmic foundations of reinforcement learning. Please join us in congratulating the two recipients! bit.ly/4hpdsbD
0
0
10
1,423
I'm giving a talk next week on our recent NeurIPS paper, "Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers". Feel free to join if you are available! :)
Thank you @CohereForAI for hosting me!
📜 arxiv.org/abs/2411.15370
Thank you @CohereForAI for hosting me!
📜 arxiv.org/abs/2411.15370
Be sure to join us next week on January 28th as we host @Gautham529 for a presentation on "Streaming Deep Reinforcement Learning."
Thanks to @rahul_narava and @gustiwinata_ for organizing this event 🌟
0
3
20
1,582
I'm in Vancouver for #NeurIPS2024. Ping me if you're around and want to chat about deep RL, robotics, continual learning and such!
0
0
8
429
Our NeurIPS paper is now on arXiv:
We introduce Action Value Gradient (AVG), a novel incremental deep RL method that learns in real-time, one sample at a time — no batch updates, target networks or a replay buffer!
Co-authors @mhmd_elsaye @bellingerc @white_martha @rupammahmood
We introduce Action Value Gradient (AVG), a novel incremental deep RL method that learns in real-time, one sample at a time — no batch updates, target networks or a replay buffer!
Co-authors @mhmd_elsaye @bellingerc @white_martha @rupammahmood
2
22
92
9,843
AVG incorporates the following normalization and scaling techniques to stabilize learning:
1⃣ Observation Normalization
2⃣ Normalization of the Penultimate Layer Feature Activations
3⃣ Scaling the Temporal Difference (TD) Error
1⃣ Observation Normalization
2⃣ Normalization of the Penultimate Layer Feature Activations
3⃣ Scaling the Temporal Difference (TD) Error
1
1
4
406
AVG is stable and offers robust performance on continuous control benchmarks, including both MuJoCo simulations and real robots — the UR5 industrial robot arm and the iRobot Roomba vacuum cleaner.
📜 Paper: arxiv.org/abs/2411.15370
💻 Code: github.com/gauthamvasan/a…
📜 Paper: arxiv.org/abs/2411.15370
💻 Code: github.com/gauthamvasan/a…
0
1
10
512
Gautham Vasan retweeted
Would you believe that deep RL can work without replay buffers, target networks, or batch updates? Our recent work gets deep RL agents to learn from a continuous stream of data one sample at a time without storing any sample. Joint work with @Gautham529 and @rupammahmood.
9
103
617
164,063
I'm presenting our work on reward design and robot learning at RLC today! We show that it is possible to learn pixel-based policies from scratch on real robots within two to three hours using only sparse rewards 🤖
📜 arxiv.org/abs/2407.00324
🎞️ youtu.be/a6zlVUuKzBc?si…
#RLC
📜 arxiv.org/abs/2407.00324
🎞️ youtu.be/a6zlVUuKzBc?si…
#RLC
0
1
4
398
Check out our research on how to specify rewards in RL:
TL;DR: Sparse reward formulations can lead to higher-quality policies and outperform dense reward policies on their own metrics.
Paper 📜: arxiv.org/abs/2407.00324
#reinforcementlearning #robotics
TL;DR: Sparse reward formulations can lead to higher-quality policies and outperform dense reward policies on their own metrics.
Paper 📜: arxiv.org/abs/2407.00324
#reinforcementlearning #robotics
1
2
5
295
Gautham Vasan retweeted
#AGIComics offers a definition of AGI that is achievable, based on economics, and provably safe.
1
8
31
4,795
Come checkout our cool poster @AAMASconf this week! 🇳🇿
Currently in Auckland, New Zealand 🇳🇿 to present #MaDi at @AAMASconf!
When: Wednesday, 8 May, 2pm
Where: Crystal Room 1
Paper, code, videos: bramgrooten.nl/madi/
0
0
3
174
Trying to put myself out there more often.
Some thoughts on Moravec's paradox, sim-to-real transfer, and robot learning: gauthamvasan.github.io/posts/simulato…
Hope you find it interesting!
#AI #Robotics
Some thoughts on Moravec's paradox, sim-to-real transfer, and robot learning: gauthamvasan.github.io/posts/simulato…
Hope you find it interesting!
#AI #Robotics
1
1
3
648
Gautham Vasan retweeted
Hope to see you at @AAMASconf 2024 in New Zealand!
Thank you to my co-authors Tristan Tomilin, @Gautham529, Matt Taylor, @rupammahmood, Meng Fang, @pechenizkiy, and Decebal Mocanu.
📜arxiv.org/abs/2312.15339
💻github.com/bramgrooten/ma…
Thank you to my co-authors Tristan Tomilin, @Gautham529, Matt Taylor, @rupammahmood, Meng Fang, @pechenizkiy, and Decebal Mocanu.
📜arxiv.org/abs/2312.15339
💻github.com/bramgrooten/ma…
0
2
5
328
Explored some practical, oft-ignored challenges in continual learning on real-world robots in a recent AI Seminar, delving into:
1️⃣ Specifying reinforcement learning tasks
2️⃣ Setting up a real-time learning agent
youtu.be/QO0mmHMJvRQ?si… 🤖
1️⃣ Specifying reinforcement learning tasks
2️⃣ Setting up a real-time learning agent
youtu.be/QO0mmHMJvRQ?si… 🤖
0
1
2
124
Thank you @neuralbertatech, for organizing the natChat 2023 event and making my invited talk public!
youtu.be/kkJ8-k-_CPQ
#reinforcementlearning #robotics
youtu.be/kkJ8-k-_CPQ
#reinforcementlearning #robotics
0
1
3
131
Gautham Vasan retweeted
A packed house to hear @BFlanaganUofA from the @UAlberta and @AmiiThinks announce that 20 new faculty will be hired in AI across campus in the next 3 years, with 5 of these positions in CS.
1
14
59
10,546