Pinned Tweet
If the end goal of robot hands is to perform human motion, then we should optimize the hardware design with human motion - and on a large scale!
We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.
We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.
9
39
240
38,417
It’s really fun to see how people are using the pi model! Sometimes we are like: wow I can’t believe that’s a potential use case 🤔
Announcing the beta release of the Pi API. We’ve partnered with companies across diverse industries to adapt our models to their robots and applications, from harvesting strawberries to assembling servers. Now we’re opening access to more teams.
1
0
25
853
yisha retweeted
Announcing the beta release of the Pi API. We’ve partnered with companies across diverse industries to adapt our models to their robots and applications, from harvesting strawberries to assembling servers. Now we’re opening access to more teams.
134
248
2,345
420,520
What’s the most important aspect that serves a good policy? If you have thoughts, please submit to our workshop!
Robot-learning folks: have you ever found that a seemingly minor system choice beneath the policy—controller tuning, hardware selection, or inference infrastructure—made a much bigger difference to your robot’s performance than an architecture or algorithm tweak? How do you ort that effect?
That’s the question behind our Workshop on Everything Beneath the Policy at #CoRL2026.
0
1
22
1,943
yisha retweeted
A video from a Pi robot deployed at Dandelion Chocolate, fully autonomous w/ no interventions. 🤖
Deploying robots has taught us surprising lessons about the gap between proof-of-concept (i.e. building one box) and real-world utility (productively building boxes for hours).
I part is building the box, since it’s the most dexterous, but that wasn't what we found. Counterintuitively, the hardest part was reliably stacking the boxes. Our original table-mounted robot had poor visibility of the stack without special separately-mounted cameras. Plus, stacking requires more generalization (each box is placed in a different location), and an imprecisely-placed box can lead the entire stack to collapse many boxes later.
We recently switched this deployment over to a mobile robot, and it's fun to watch the robot being completely self sufficient for multiple hours. 🙂 Data and feedback from real world use-cases like these are quite valuable for the π pre-trained model as we scale!
Deploying robots has taught us surprising lessons about the gap between proof-of-concept (i.e. building one box) and real-world utility (productively building boxes for hours).
I part is building the box, since it’s the most dexterous, but that wasn't what we found. Counterintuitively, the hardest part was reliably stacking the boxes. Our original table-mounted robot had poor visibility of the stack without special separately-mounted cameras. Plus, stacking requires more generalization (each box is placed in a different location), and an imprecisely-placed box can lead the entire stack to collapse many boxes later.
We recently switched this deployment over to a mobile robot, and it's fun to watch the robot being completely self sufficient for multiple hours. 🙂 Data and feedback from real world use-cases like these are quite valuable for the π pre-trained model as we scale!
46
67
923
106,174
gripper + 3 finger = 5 finger hand is kind of cute 🥺 congrats on the release! @yifengzhu_ut @zipengfu
Introducing OM-1, our first robot foundation model, zero-shot generalizing to any robot: table-top arms, industrial arms and humanoids.
- learned directly from human manipulation data
- no teleop/robot data
- close to human-level dexterity and efficiency
- multi-robot collab
2
0
20
1,749
yisha retweeted
My group at Johns Hopkins now has a name and a website
Introducing the Brains, Bots, and Behavior Lab
b3.cs.jhu.edu
We're always looking for creative researchers at all levels to join us!
Introducing the Brains, Bots, and Behavior Lab
b3.cs.jhu.edu
We're always looking for creative researchers at all levels to join us!
15
63
565
46,870
yisha retweeted
Children learn to walk by falling. Robots fail too—but we usually throw those failures away.
Yet failed rollouts tell us something successful demonstrations cannot: what bad actions actually lead to.
So, can robots learn from their failures too?
Introducing FACT: Failure-Aware ld-Action Models.
🔗website: fact-wam.github.io
📄arxiv: arxiv.org/abs/2608.10232
💻 code: github.com/Bariona/FACT
Yet failed rollouts tell us something successful demonstrations cannot: what bad actions actually lead to.
So, can robots learn from their failures too?
Introducing FACT: Failure-Aware ld-Action Models.
🔗website: fact-wam.github.io
📄arxiv: arxiv.org/abs/2608.10232
💻 code: github.com/Bariona/FACT
15
41
321
76,210
I’ve been joking with friends that my main job this year has been interviewing 🫠
Now I’m finally excited to share what’s next: I’ll be joining NYU as an Assistant Professor next fall, and before that, I’ll be spending a year at @physical_int !
I feel incredibly fortunate to years at UCSD working on control-hardware co-design.
When I was looking for a postdoc, I told people I wanted to work on robot learning for hardware design. I’m deeply grateful that @xiaolonw and Mike trusted me enough to start something none of us quite knew how to do.
Whenever I mentioned working with both of them, people either knew only one of them (they come from very different communities), or found the combination surprising. I find it to be really fun and I learnt so much more than I would anywhere else.
Although I think jointly optimizing policies and hardware at scale is still somewhat early, I strongly believe in optimizing (or at least iterating) hardware alongside policy training.
Over the past few decades, we’ve developed a lot of depth in individual hardware components: sensors, actuators, mechanisms, materials, etc. But we still haven’t explored enough of their combinations as integrated systems, or understood their actual usefulness for policy.
As coding agents get more and more powerful, learning where and how hardware is bottlenecking robots is really valuable. And the only way to understand this is to scale up and try things :)
I’m getting ready to join the bay area robotics community, but it’s still a bit sad to leave the beautiful San Diego.
I had a lot of fun sampling taco shops on Tuesdays (although I eventually converged to two places on my way home...). I will definitely miss the on campus climbing gym, and the forever-sunny-and-warm outdoor swimming pool next to it 🥺
Now I’m finally excited to share what’s next: I’ll be joining NYU as an Assistant Professor next fall, and before that, I’ll be spending a year at @physical_int !
I feel incredibly fortunate to years at UCSD working on control-hardware co-design.
When I was looking for a postdoc, I told people I wanted to work on robot learning for hardware design. I’m deeply grateful that @xiaolonw and Mike trusted me enough to start something none of us quite knew how to do.
Whenever I mentioned working with both of them, people either knew only one of them (they come from very different communities), or found the combination surprising. I find it to be really fun and I learnt so much more than I would anywhere else.
Although I think jointly optimizing policies and hardware at scale is still somewhat early, I strongly believe in optimizing (or at least iterating) hardware alongside policy training.
Over the past few decades, we’ve developed a lot of depth in individual hardware components: sensors, actuators, mechanisms, materials, etc. But we still haven’t explored enough of their combinations as integrated systems, or understood their actual usefulness for policy.
As coding agents get more and more powerful, learning where and how hardware is bottlenecking robots is really valuable. And the only way to understand this is to scale up and try things :)
I’m getting ready to join the bay area robotics community, but it’s still a bit sad to leave the beautiful San Diego.
I had a lot of fun sampling taco shops on Tuesdays (although I eventually converged to two places on my way home...). I will definitely miss the on campus climbing gym, and the forever-sunny-and-warm outdoor swimming pool next to it 🥺
51
22
473
79,461
Yaqi is super kind and sweet 🥹whoever join her lab will have a happy phd life 😉
Very excited to share that I'll be joining UIUC ECE and CS (affiliate) as an Assistant Professor @UofIllinois
I'm recruiting PhD students, interns, and postdocs to build trustworthy AI agents and robots that reason, learn, and grow alongside humans. If you're interested in reach out at yaqixie.ai@gmail.com
Many thanks to my advisors and collaborators for all their help and support! @haroldsoh @ Katia_Sycara @RamananDeva @ Alexander_Gray @ David_Hsu @ksmeel @ Mohan_Kankanhalli @joecampb @SimonStepputtis @ Woojun_Kim @yswhynot
1
5
183
34,337
Touch has become a key ingredient for dexterous robot manipulation, but which tactile sensor should you actually use?
We're excited to introduce TacO, a benchmark for evaluating tactile sensors across real-world manipulation tasks.
🌮 TacO Benchmark provides:
Cross-modality sensors (vision-, magnetic-, acoustic-, and resistive-based tactile sensing)
Standardized imitation learning framework with all kinds of tactile sensors
Open-source code, data, and hardware
Our experiments show that there is no universally best tactile sensor😌(even the expensive ones). The right choice depends on the tasks. We hope TacO helps the community build and evaluate the next generation of tactile sensors and how to use them to improve vision-based robot policies.
We're excited to introduce TacO, a benchmark for evaluating tactile sensors across real-world manipulation tasks.
🌮 TacO Benchmark provides:
Cross-modality sensors (vision-, magnetic-, acoustic-, and resistive-based tactile sensing)
Standardized imitation learning framework with all kinds of tactile sensors
Open-source code, data, and hardware
Our experiments show that there is no universally best tactile sensor😌(even the expensive ones). The right choice depends on the tasks. We hope TacO helps the community build and evaluate the next generation of tactile sensors and how to use them to improve vision-based robot policies.
6
19
84
6,647
If you are interested in the commercial sensors, here are the detailed links!
FSR: a.co/d/05paXSTR or similar
Contact mic: a.co/d/06kEZqlT, sound card: a.co/d/030iYREj (similar ones should work too)
eflesh & flexitac: details in paper, egain: customized
FSR: a.co/d/05paXSTR or similar
Contact mic: a.co/d/06kEZqlT, sound card: a.co/d/030iYREj (similar ones should work too)
eflesh & flexitac: details in paper, egain: customized
0
0
1
341
We also thank again @binghao_huang and @venkyp2000 for donating sensors 😆, and our friends at SNU for making and shipping eGain all the way across the ocean. We won't be able to play with so many sensors without their help!
1
0
3
842
This is led by @anyazorin , collaborating with Zilin Si, Myungsun Park, @jnsungp , @ABuynitsky , @bhadang_sachin , Taejun Park, Sohee John Yoon, Yong-Lae Park, Oliver Kroemer, Zeynep Temel, Michael T. Tolley, @xiaolonw
0
0
1
467
Project page: tacobench.github.io
Code: github.com/TacObench/TacO
Hardware: github.com/TacObench/TacO…
Paper: arxiv.org/abs/2605.21976
Code: github.com/TacObench/TacO
Hardware: github.com/TacObench/TacO…
Paper: arxiv.org/abs/2605.21976
0
0
3
507
One of the best PhD defenses I’ve ever seen 🥺 I’m so jealous whoever that gets to work with Huy 🥺
Last week, I defended my PhD dissertation, "Robots From Anyone: Shaping Behaviors and Embodiments with Handheld Grippers". You can find the talk here!
youtu.be/Zlmdzv4rYzM
0
0
3
1,765
yisha retweeted
RL training for LLMs involves exposure to problems in the “Goldilocks zone” of difficulty: not too hard, not too easy.
But difficulty is not the only thing that matters.
Problem type matters, too.
And LLMs do not see or organize problem types the way humans do.
This is the old Bandits.
The fundamental issue is this: when using RL to train LLMs, there will always be problems that are more productive for the policy at a given training iteration. Across a dataset, there are an astronomical number of possible batch orderings; uniform sampling is unlikely to consistently land on the most useful ones by chance.
This is why many adaptive curriculum learning and difficulty-aware methods focus on the policy’s “optimal learning zone,” avoiding examples that are already trivial or still impossible for the LLM to solve.
But problem type is usually treated differently.
Task decompositions within a training set are often manually defined according to human semantics, or ignored entirely, with an entire dataset/domain treated as “one task,” despite the immense heterogeneity seen even in small training datasets.
It is generally understood that LLMs may not share our concept of difficulty. But it seems less recognized that LLMs may not share our concept of problem type, either. In both cases, training the LLM according to our own semantics can obscure fine-grained learning dynamics that could otherwise be exploited.
In this work, we seek to derive an adaptive curriculum learning method that caters not only to the policy’s ability, but also its perception: its latent organization of tasks.
Driven by the manifold hypothesis and the Bitter Lesson, we leverage computation to derive curriculum structure from the policy’s latent geometry, then use Bayesian inference over that structure to guide search and learning over the training set. The result is Bayesian Manifold Curriculum (BMC): an algorithm that does not just try to find problems of the right difficulty, but instead orchestrates training effort across diverse and interacting problem types.
More technically, we frame problem sampling for LLMs as a manifold-structured bandit problem with endogenous non-stationarity. The tl;dr is that the problems/arms are diverse, structured, and interact through policy updates, so the typical non-stationary bandit framing is not quite the right fit for training LLMs. BMC is derived from this new framing.
🌐Website: darrienmckenzie.com/manifold-bandi…
📰Paper: arxiv.org/abs/2606.19750
💻Code: github.com/DarrienMcKenzi…
🧵1/N
But difficulty is not the only thing that matters.
Problem type matters, too.
And LLMs do not see or organize problem types the way humans do.
This is the old Bandits.
The fundamental issue is this: when using RL to train LLMs, there will always be problems that are more productive for the policy at a given training iteration. Across a dataset, there are an astronomical number of possible batch orderings; uniform sampling is unlikely to consistently land on the most useful ones by chance.
This is why many adaptive curriculum learning and difficulty-aware methods focus on the policy’s “optimal learning zone,” avoiding examples that are already trivial or still impossible for the LLM to solve.
But problem type is usually treated differently.
Task decompositions within a training set are often manually defined according to human semantics, or ignored entirely, with an entire dataset/domain treated as “one task,” despite the immense heterogeneity seen even in small training datasets.
It is generally understood that LLMs may not share our concept of difficulty. But it seems less recognized that LLMs may not share our concept of problem type, either. In both cases, training the LLM according to our own semantics can obscure fine-grained learning dynamics that could otherwise be exploited.
In this work, we seek to derive an adaptive curriculum learning method that caters not only to the policy’s ability, but also its perception: its latent organization of tasks.
Driven by the manifold hypothesis and the Bitter Lesson, we leverage computation to derive curriculum structure from the policy’s latent geometry, then use Bayesian inference over that structure to guide search and learning over the training set. The result is Bayesian Manifold Curriculum (BMC): an algorithm that does not just try to find problems of the right difficulty, but instead orchestrates training effort across diverse and interacting problem types.
More technically, we frame problem sampling for LLMs as a manifold-structured bandit problem with endogenous non-stationarity. The tl;dr is that the problems/arms are diverse, structured, and interact through policy updates, so the typical non-stationary bandit framing is not quite the right fit for training LLMs. BMC is derived from this new framing.
🌐Website: darrienmckenzie.com/manifold-bandi…
📰Paper: arxiv.org/abs/2606.19750
💻Code: github.com/DarrienMcKenzi…
🧵1/N
3
38
208
11,375
yisha retweeted
Train on one tactile and deploy on another
A robot policy trained on one tactile sensor usually breaks the moment you swap the sensor. New hardware → recollect data → retrain.
We built TACTX: train tactile policies on one sensor, deploy zero-shot on a physically different one.
The catch, these sensors don't just look contact through totally different physics:
👁️ Daimon — vision-based · 🧲 eFlesh — magnetic · ⚡ FlexiTac — resistive
Same contact but completely different raw signals. 🧵
0
2
24
5,529
We show that the policy trained with one tactile sensor can be transferred to another sensor! Also, great thanks to @binghao_huang and @venkyp2000 for donating sensors :D
A robot policy trained on one tactile sensor usually breaks the moment you swap the sensor. New hardware → recollect data → retrain.
We built TACTX: train tactile policies on one sensor, deploy zero-shot on a physically different one.
The catch, these sensors don't just look contact through totally different physics:
👁️ Daimon — vision-based · 🧲 eFlesh — magnetic · ⚡ FlexiTac — resistive
Same contact but completely different raw signals. 🧵
0
2
22
4,372
This so fun to play with!
Super excited to share the last paper of my PhD: "Hallucination in World Models is Predictable and Preventable"✨
We train a 350M-param generative world model on a large dataset w/ 210 tasks and show that we can predict *when* hallucination happens and use that to fix it!
🧵1/n
1
0
2
1,111