Pinned Tweet
Introducing GEN-1.5, a one-shot learner.
It can learn new tasks in a few seconds. Show it what to do, and it generalizes.
This capability emerged from pretraining on physical data at scale, as a step towards our mission of building general intelligence for the physical world.
It can learn new tasks in a few seconds. Show it what to do, and it generalizes.
This capability emerged from pretraining on physical data at scale, as a step towards our mission of building general intelligence for the physical world.
324
1,682
12,172
3,441,805
Generalist retweeted
.@GeneralistAI CEO @peteflorence has been watching robot arms make the same kind of mistake: halfway through a task, they meander off to the side in a way they shouldn’t.
With a weak base model, he said, fixing it feels like playing Whac-A-Mole; every failure you patch makes But that has changed as the base models have gotten stronger. Now, Pete recalled, you only have to give a “very small amount of guidance” on what not to do, and “it just won’t do it again.”
With a weak base model, he said, fixing it feels like playing Whac-A-Mole; every failure you patch makes But that has changed as the base models have gotten stronger. Now, Pete recalled, you only have to give a “very small amount of guidance” on what not to do, and “it just won’t do it again.”
"If you're not touching and moving the world and you're a robot, what are you doing?"
@peteflorence is co-founder and CEO of @GeneralistAI, a company building general-purpose intelligence for the physical world through robotics foundation models. In mid-August, Generalist atest model, which Anthropic researcher Sholto Douglas has described as a "GPT-3 moment for robotics".
Florence's central claim is that dexterity, touching and manipulating the physical world, is the robotics equivalent of coding for language models: a broad capability with the potential to create a self-improvement loop in which robots build robots.
(2:24) GEN-1.5 and one-shot learning
(10:53) Dexterity in robotics
(27:38) Generalist’s founding thesis and what sets it apart
(32:17) Protecting simplicity
(42:15) Balancing general intelligence with commercial deployment
(50:52) Geopolitics, safety, and alignment
(1:03:37) Book recommendations
0
3
11
2,751
Our CEO Pete Florence on The Generalist podcast, talking all things robots and how we're building for the physical world.
@mariogabriele @thegeneralistco
Listen to the whole conversation - link in the comments below
@mariogabriele @thegeneralistco
Listen to the whole conversation - link in the comments below
2
3
35
3,824
How many pucks could a puck bot pack if a puck bot could pack pucks? 🏒
12 hockey pucks is a tight fit in this container – watch the model fix the arrangement when pucks get stuck or misplaced.
GEN-1.5 doesn't just puck around. Read more about it in our blog post and sign up comments below.
12 hockey pucks is a tight fit in this container – watch the model fix the arrangement when pucks get stuck or misplaced.
GEN-1.5 doesn't just puck around. Read more about it in our blog post and sign up comments below.
9
12
115
16,974
Other tasks our models can do:
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
0
0
4
1,633
GEN-1.5 puts together a kit of cosmetics, then puts together another kit, then another kit, then another...
We're expanding access to GEN-1.5
Sign up to get in line
generalistai.com/waitlist
We're expanding access to GEN-1.5
Sign up to get in line
generalistai.com/waitlist
7
10
173
12,859
Other tasks our models can do:
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
0
0
3
2,001
GEN-1.5 packs a bottle into a box, then packs another bottle into a box, then another bottle, then another one, then another...
Read more about GEN-1.5 in our blog post in the comments below.
Read more about GEN-1.5 in our blog post in the comments below.
9
25
331
49,141
Read more about GEN-1.5, our latest foundation model for the physical world: generalistai.com/blog/gen-1.5
1
1
11
3,361
Other tasks our models can do:
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
0
0
1
680
Robot puts a ring on it
GEN-1.5 puts an O-ring on a hydraulic rod gland, then takes the ring off, then puts the ring back on, then takes the ring off, then ring on, ring off, ring on...
Read more about GEN-1.5 in our blog post in the comments below.
GEN-1.5 puts an O-ring on a hydraulic rod gland, then takes the ring off, then puts the ring back on, then takes the ring off, then ring on, ring off, ring on...
Read more about GEN-1.5 in our blog post in the comments below.
22
43
521
53,736
Other tasks our models can do:
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
0
0
8
4,988
A simple example of physical prompt steerability: same environment, different prompts, different behaviors.
Thanks @JagdeepBhatia8 for the suggestion.
Read more about GEN-1.5 in our blog post in the comments below.
Thanks @JagdeepBhatia8 for the suggestion.
Read more about GEN-1.5 in our blog post in the comments below.
Physical prompting is an elegant idea!
One request for the @GeneralistAI team: can we see an example of *physical prompt steerability*, where different prompts induce distinct behavior in the *same* environment?
I'm curious whether GEN-1.5 is listening to the physical prompt or simply executing the most likely action sequence for the scene under its pretraining prior.
8
9
118
23,563
@JagdeepBhatia8 Other tasks our models can do:
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
youtube.com/playlist?list=…
Read more about GEN-1.5, our latest foundation model for the physical world:
generalistai.com/blog/gen-1.5
1
0
6
4,029
We've reduced the time it takes to go from physical prompt → robot behavior.
The faster anyone can teach a robot to do something new, the easier it becomes to scale physical work.
Read more about GEN-1.5 in our blog post in the comments below.
The faster anyone can teach a robot to do something new, the easier it becomes to scale physical work.
Read more about GEN-1.5 in our blog post in the comments below.
8
27
239
27,615
Read more about GEN-1.5, our latest foundation model for the physical world: generalistai.com/blog/gen-1.5
0
0
18
4,170
Generalist retweeted
Last week @willknight came by and got a preview of one-shot prompting and we had some fun improvisational moments with the robot. Thanks Will for helping capture the moment.
I'm usually wary about talk of "the ChatGPT moment for robots" because language is less complex than the physical world. After visiting @GeneralistAI, though, where I saw their robots perform impressive manipulation with just a prompt, I think you can see it happening in a limited way. (Thanks to @peteflorence and co for having me!) wired.com/story/generali…
2
3
42
9,333
Generalist retweeted
Good question thanks Ani. Here's a video that gives a sense of the answer here about the effect of prompting:
- in the first part, the model is just babbling with no prompt in context
- at the end, we add a prompt for taking money out of a pouch (generalizing to the wallet)
- in the first part, the model is just babbling with no prompt in context
- at the end, we add a prompt for taking money out of a pouch (generalizing to the wallet)
@peteflorencek@andyzengineerrious if you've tried an extreme version of this: no in-context example at all; just place objects in front of the robot and see if it can infer the task.
I suspect this will have non-trivial success rates, which would also allow you to figure out how much of the benefit is actually coming from the prompt (rather than pure zero-shot task inference + capability).
I remember Russ showed something like this in his Stanford talk on LBMs last year:
youtu.be/TN1M6vg4CsQ?si…
16
28
271
33,372
Generalist retweeted
We started planning yesterday’s @GeneralistAI GEN-1.5 announcement about 3 weeks ago. Two weeks ago, already feeling whiplash from new results, I asked ChatGPT to create this image to share in our internal meme channel.
3
6
42
4,390