Yevgen Chebotar retweeted
Exactly! Tool use adds another layer of dexterity and sequencing. Here's an uncut end-to-end policy using a drill to assemble a toy car. Notice how the robot makes micro-adjustments to the drilling angle and still completes the fastening even when the object isn't perfectly d of robustness we're excited about. 🦾🔩
My morning ritual: coffee, then watch our robot assemble stuff. The model isn't fast, but it measures every grasp, obsesses over every alignment, and handles every piece like an heirloom. Watching is therapeutic, even meditative. Simple pleasure from the execution of a task well p, end-to-end policy in one go.
3
8
45
5,209
Long visuomotor context is one of THE most important missing parts for robotic foundation models, there is no way around it if we want generic policies that remember more than a couple of camera frames. However, adding visual context naively significantly increases inference time on the robot and doesn’t scale beyond very short horizons.
In RoboTTT, we found a way to scale it to orders of magnitude more to up to 8k timesteps without a significant increase of inference time through a principled integration of Test-Time-Training (TTT) into the foundation models. The context is saved in fast weights of the model that can be updated at inference in constant time.
This opens a range of new applications and possibilities:
- Performance on long-horizon dexterous tasks keeps on increasing with context length. It doesn’t require any separate planner for coordinating the subtasks, can self-correct and becomes robust to perturbations.
- One-shot imitation from human videos can be done in-context by just concatenating human videos and robot trajectories.
- We can develop novel in-context improvement algorithms, such as DAgger distillation that meta-learns to map suboptimal robot actions to human corrections within the same sequence.
Check out the deep dive and the paper for more details!
research.nvidia.com/labs/gear/robo…
In RoboTTT, we found a way to scale it to orders of magnitude more to up to 8k timesteps without a significant increase of inference time through a principled integration of Test-Time-Training (TTT) into the foundation models. The context is saved in fast weights of the model that can be updated at inference in constant time.
This opens a range of new applications and possibilities:
- Performance on long-horizon dexterous tasks keeps on increasing with context length. It doesn’t require any separate planner for coordinating the subtasks, can self-correct and becomes robust to perturbations.
- One-shot imitation from human videos can be done in-context by just concatenating human videos and robot trajectories.
- We can develop novel in-context improvement algorithms, such as DAgger distillation that meta-learns to map suboptimal robot actions to human corrections within the same sequence.
Check out the deep dive and the paper for more details!
research.nvidia.com/labs/gear/robo…
We scaled robot policies to 8K timesteps of visuomotor context, orders of magnitude beyond current SoTAs, at constant inference latency.
Introducing RoboTTT 🤖
With minutes of experience in context, our robots:
🎥 one-shot imitate human video demos
📈 improve themselves during rom perturbations
⚙️ complete a 5-minute, 10-stage assembly end to end
🌐 research.nvidia.com/labs/gear/robo…
Dive in 🧵
2
4
34
181,709
Yevgen Chebotar retweeted
Proud to introduce EgoScale: We pretrained a GR00T VLA model on 20K+ hours of egocentric human video and discovered that robot dexterity can be scaled, not with more robots, but with more human data. A thread on 🧵what we learned. 👇
26
65
333
103,057
Yevgen Chebotar retweeted
SONIC is now open-source!
Generalist whole-body teleoperation for EVERYONE!
Our team has long been building comprehensive pipelines for whole-body control, kinematic planner, and teleoperation, and they will all be shared.
This will be a continuous update; inference code + aining code and gr00t integration coming soon!
Code: github.com/NVlabs/GR00T-W…
Docs: nvlabs.github.io/GR00T-WholeBod…
Site: nvlabs.github.io/GEAR-SONIC/
Generalist whole-body teleoperation for EVERYONE!
Our team has long been building comprehensive pipelines for whole-body control, kinematic planner, and teleoperation, and they will all be shared.
This will be a continuous update; inference code + aining code and gr00t integration coming soon!
Code: github.com/NVlabs/GR00T-W…
Docs: nvlabs.github.io/GR00T-WholeBod…
Site: nvlabs.github.io/GEAR-SONIC/
36
197
930
262,379
We are moving towards world-model based backbones for robotic policies, pre-trained on web-scale videos and outputting robotic actions directly within the same diffusion model!
There are new levels of transfer to unseen tasks and motions being unlocked, something that has been missing in VLM-based pre-training ever since the early days of RT-2 VLA models, which although generalizing well to new objects and semantics always struggled with new “verbs” and “motions”. We also observe signs of efficient cross-embodiment transfer to new robots with a little amount of data.
There are a lot of optimizations that allow us to run a 14B world action model in real time! My favorite trick is DreamZero-Flash, which while still denoising videos and actions jointly, introduces separate noise schedules, such that actions can be denoised much faster than videos to enable higher frequency control!
Check out the website, paper and especially the eval gallery!
dreamzero0.github.io
There are new levels of transfer to unseen tasks and motions being unlocked, something that has been missing in VLM-based pre-training ever since the early days of RT-2 VLA models, which although generalizing well to new objects and semantics always struggled with new “verbs” and “motions”. We also observe signs of efficient cross-embodiment transfer to new robots with a little amount of data.
There are a lot of optimizations that allow us to run a 14B world action model in real time! My favorite trick is DreamZero-Flash, which while still denoising videos and actions jointly, introduces separate noise schedules, such that actions can be denoised much faster than videos to enable higher frequency control!
Check out the website, paper and especially the eval gallery!
dreamzero0.github.io
Introducing DreamZero 🤖🌎 from @nvidia
> A 14B “World Action Model” that achieves zero-shot generalization to unseen tasks & few-shot adaptation to new robots
> The key? Jointly predicting video & actions in the same diffusion forward pass
Project Page: dreamzero0.github.io
🧵 (1/10)
2
1
26
1,964
Excited to join the NVIDIA GEAR team to help build the next generation of open robotic foundation models!
13
2
152
13,987
We've made great progress on Vision-Language-Action Models for humanoids in our new Helix model!
Check out the technical report for more details:
figure.ai/news/helix
Check out the technical report for more details:
figure.ai/news/helix
Meet Helix, our in-house AI that reasons like a human
Robotics won't get to the home without a step change in capabilities
Our robots can now handle virtually any household item:
6
4
92
8,174
The path to VLAs lies through VLMs. A very nice intro for everyone interested in working with Vision-Language Models:
arxiv.org/abs/2405.17247
arxiv.org/abs/2405.17247
1
17
66
6,305
Congrats everyone, 170+ authors and contributors, great to see the robotic field coming together!
Our OpenX paper won best paper at ICRA! Congrats to all my co-authors! 🎉🎉
This is an ongoing effort, we recently added new datasets from the community that double the size of the OpenX dataset -- keep 'em coming! :)
Check datasets & how to contribute: robotics-transformer-x.github.io
0
1
23
2,851
Some personal updates!
Excited to join the team @Figure_robot to help building AI for the robot age! 🤖
Excited to join the team @Figure_robot to help building AI for the robot age! 🤖
24
1
164
66,412
Turns out classification loss works surprisingly well for value-based RL, also some nice gains when used with Q-Transfomer!
Super simple code change to get value-based deep RL scale *much* better w/ big models across the board on Atari games, robotic manipulation w/ transformers, LLM + text games, & even Chess!
Just use classification loss (i.e., cross entropy), not MSE!!
arxiv.org/abs/2403.03950🧵⬇️
0
3
16
5,820
RT-H learns a hierarchy all the way from high-level tasks through low-level “language motions” to robot actions!
✅ Improved performance and generalization through better data sharing
✅ Automated grounded “bottom-up” labeling
✅ Ability to intervene and correct with language
✅ Improved performance and generalization through better data sharing
✅ Automated grounded “bottom-up” labeling
✅ Ability to intervene and correct with language
Is language capable of representing low-level *motions* of a robot?
RT-Hierarchy learns an action hierarchy using motions described in language, like “move arm forward” or “close gripper” to improve policy learning.
📜: arxiv.org/abs/2403.01823
🏠: rt-hierarchy.github.io (1/10)
1
3
36
4,661
Yevgen Chebotar retweeted
Had a great time today with @YevgenChebotar and @QuanVng visiting @USCViterbi to give a talk on “Robot Learning in the Era of Foundation Models”. Slides out soon, packed with works from *just the past 5 months* 🤯
Thanks to @daniel_t_seita for hosting!
Thanks to @daniel_t_seita for hosting!
1
3
58
5,593
Pictures taken at RT-2 poster at @DannyDriess requests ; )
@YevgenChebotar
We miss you @TianheYu
CC @hausman_k
0
2
24
3,832
Yevgen Chebotar retweeted
How many @CSatUSC grads does it take to create a breakthrough robot? Three, apparently!
Congrats @YevgenChebotar @hausman_k and @ryancjulian who worked on Google DeepMind's revolutionary RT-2 AI model.
Find out more ⤵️
viterbischool.usc.edu/news/2023/10/t…
@gauravsukhatme @USCViterbi
Congrats @YevgenChebotar @hausman_k and @ryancjulian who worked on Google DeepMind's revolutionary RT-2 AI model.
Find out more ⤵️
viterbischool.usc.edu/news/2023/10/t…
@gauravsukhatme @USCViterbi
0
3
21
3,756
Exciting times for Robot Learning! 60 datasets from 22 different robots and 21 institutions combined in a single Open-X Embodiment data repository, resulting in over 1 million episodes and improved RT-X models! Amazing and a very important collaboration across the world! 🤖🌐
RT-X: generalist AI models lead to 50% improvement over RT-1 and 3x improvement over RT-2, our previous best models. 🔥🥳🧵
Project website: robotics-transformer-x.github.io
0
1
12
1,438
Offline RL strikes back! In our new Q-Transformer paper, we introduce a scalable framework for offline reinforcement learning using Transformers and autoregressive Q-Learning to learn from mixed-quality datasets!
Website and paper: q-transformer.github.io 🧵
Website and paper: q-transformer.github.io 🧵
7
103
516
210,693
Our real robot policies significantly improve upon RT-1 and other baselines when trained on limited amount of human demonstrations by leveraging autonomously collected negatives and dynamic programming properties of Q-learning.
1
0
15
2,119
Joint work with @QuanVng, @AlexIrpan, @hausman_k, @xf1280, @Yao__Lu, @aviral_kumar2, @TianheYu, @AlexHerzog001, @KarlPertsch, @keerthanpg, @julianibarz, @ofirnachum, @Kanishka_Rao, @chelseabfinn, @svlevine
1
1
9
1,860