Pinned Tweet
This was work done last year, but my lazy ass uploaded it just this week
It was huge fun working on this project.
It was huge fun working on this project.
yesterday, @cloneofsimo and I released a tech report on the video pretraining research we worked on last year. we believe sharing our experience with the community building video models will be of great help!
arxiv.org/abs/2603.00173
1
0
91
47,231
I agree with the general claim but the example seems to be completely hillclimbable tbh, because LLMs are extremely good at hillclimbing being likeable in general. current LLMs already show pretty sycophatic behaviors and thats the improved version... Remember 4o?
Difficult part might be being *secretly* sycophatic, which by definition, we actually wouldnt notice if its already the case with current opus models (I suspect it is)
Difficult part might be being *secretly* sycophatic, which by definition, we actually wouldnt notice if its already the case with current opus models (I suspect it is)
"this waitress is better at subtly flirting with her customers and therefore makes more in tips" and "this secretary is better at massaging the ego of her boss than the other one" and "this streamer manages to form slightly better parasocial relationships with his audience despite being kinda disgusting, but it works" is kinda difficult to hillclimb in an RLVR env, actually.
You can collect data to finetune on from the most successful examples of basically everything humans do, maybe even turn some of those into gold datasets and do preference optimisation, but that's not the same thing as being verifiable in the way code or math is
1
0
8
1,147
Whats do u think is the true asymptote for matmul?
A. 2.25 + eps
B. 2 + eps
C. 2 + c for some c < .25
D. 2
E. 2.25
Matmul is the most fundamental thing in the universe, I refuse to believe its A C E
A. 2.25 + eps
B. 2 + eps
C. 2 + c for some c < .25
D. 2
E. 2.25
Matmul is the most fundamental thing in the universe, I refuse to believe its A C E
10
0
35
7,733
Wait it doesnt have to stop here... low poly of the entire earth ... google earth but its like videogame visuals... who is making this?
Introducing Isometric SF A pixel-art map of the entire SF Bay Area, over 1,300 square miles, all painted by Klein, a 4B-parameter image-to-image model, on @modal.
Explore the map now!
isometricsf.com
4
3
85
8,694
Jesus
TLDR: they use LightningDiT (DiT+RoPE+RMS norm+SwiGLU FFN) as the backbone but compare against vanilla DiT/SiT, so it's hard to tell whether EBMs really scale better or it's just the stronger architecture.
Thanks @XiaoyuW9401147 for spotting this.
(Table from the LightningDiT 8Jx42lm)
0
1
68
7,211
I think the lesson is more like: RL is cherry on top if the cake is distillation from other frontier models (which is like nothing wrong / illegal, but you know, its not exactly SFT)
One lesson from this launch: climb as far as you can with SFT before relying on expensive RL.
RL is still the cherry on top, as @ylecun once put it.
1
7
139
10,677
Interesting stuff
SphereEncoder2 is out! Our 2nd-gen ImageNet model achieves SOTA image quality (FDr6 score) among all known pixel-space image generators, including diffusions.
The Sphere Encoder is a faster alternative to diffusion that produces images in a single forward pass. It's basically ids that maps 100k-dimensional random latent vectors directly to natural images. Here's some single-pass images from a model trained only on ImageNet-1K.
0
0
26
3,185
Fal is truly generational company. Incredible leadership and talents. Congrats friends.
Today, we’re announcing we raised our Series E & F rounds for a combined total of $1.65B. Our Series E was led by @sequoia and our Series F was led by @BessemerVP with continued investment from @nvidia.
We’re excited to push the boundaries of what’s possible even more next year and e the team to build the way we want to.
8
3
87
5,066
Friend i made in military camp harvests honey, and his honey is really good!
100% natural, organic, japanese snowball flower fermented honey, produced in Ji-ri-mountain area in Korea
130$ for 2.4kg (5.29 lb)
Im trying to convince him to ship overseas because i think there might is!! If u are interested u can dm me
100% natural, organic, japanese snowball flower fermented honey, produced in Ji-ri-mountain area in Korea
130$ for 2.4kg (5.29 lb)
Im trying to convince him to ship overseas because i think there might is!! If u are interested u can dm me
0
0
26
2,624
Being really sick makes me so greatful for the days not sick
But when im not sick i dont feel same level of greatfulness
Always do everything you can for the best immune system folks
But when im not sick i dont feel same level of greatfulness
Always do everything you can for the best immune system folks
1
0
16
1,433
Completely randomly it occured to me : Why did human race never tried to 'evolve' monkeys? like, artifical crossing to get the smartest monkey? if we have a hyper-automated monkey farm that we push our maximal effort to get fastest-growing, smartest monkey, could we have gotten e monkey?
It is obviously super immoral but you know, to non-vegetarian, its weird there wasnt major effort to do this while we are doing this to cows.
It is obviously super immoral but you know, to non-vegetarian, its weird there wasnt major effort to do this while we are doing this to cows.
24
0
30
5,314
Ok i can use my phone in the weekend
So what the fuck happened this week why is my portfolio fucked up
So what the fuck happened this week why is my portfolio fucked up
2
0
21
3,369
Honestly Gary is just chad. When everyone is amazed, excited, bittersweet, awed, slightly scared of the technological progress of LLMs, he has the bulls to go "nah this aint shyt"
since many people are asking me about the new math results from OpenAI, here is my take.
1. It uses symbolic AI (Lean, etc) in addition to LLMs which is what i said for years we would need to do; this this confirms what I actually said (as opposed to various fictitious misrepresentations that run rampant around here )
2. It still doesn’t mean they have AGI; the math stuff AFAIK does not generalize broadly, because you can’t use the same symbolic verification (and symbolic data augmentation techniques) in the open-ended real world.
So yes I do feel vindicated wrt to neurosymbolic AI but no the G in AGI (general) is not yet solved.
What I said in 2019 remains true: neurosymbolic AI is necessary but not sufficient; world models are also key.
4
1
58
5,960
It has to be some gods endgame plot that latest theoretical breakthrough on gradient descent and matrix multiplication came from
Gradient descent and matrix multiplication
Gradient descent and matrix multiplication
9
106
1,504
28,935
Context: LLM resolved latest theoretical bound for matmul and gradient descent
The Silver Rate Is Tight for GD!
After non-trivial interaction with ChatGPT 6, we prove the lower bound n^{-(p+C\sqrt{\log\log n/\log n})}, p=\log_2(1+\sqrt{2}), for GD with predetermined nonnegative stepsizes in smooth convex optimization. The silver exponent is optimal.
0
4
59
3,692
LLM went from hallucinating 3-digit multiplication to fundamentally improving multiplication algorithm itself.
...In 5 years.
...In 5 years.
oh yeah btw guys we can do integer multiplication faster than n log n lol
I was definitely very surprised when this one came in lol
github.com/openai/math/tr…
29
147
2,351
55,114
its kinda bittersweet tbh
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
4
0
51
3,348