GPU MODE retweeted
A very interesting blog post by @EliaIsPosting about applying autoresearch to a @GPU_MODE competition, winning 3rd place!
Setup includes an orchestrator + subagents: idea generation, workers to implement the ideas, redteamers to audit the conclusions.
Measuring actual signal e is obviously a major challenge, which is highlighted in this blog post. The solution was an escalating evaluation protocol, which brings the comparison floor to ~0.15%.
Link to blog post: elianaive.com/posts/another-…
Setup includes an orchestrator + subagents: idea generation, workers to implement the ideas, redteamers to audit the conclusions.
Measuring actual signal e is obviously a major challenge, which is highlighted in this blog post. The solution was an escalating evaluation protocol, which brings the comparison floor to ~0.15%.
Link to blog post: elianaive.com/posts/another-…
7
6
39
7,415
GPU MODE retweeted
oh shit i am number 1 on hackernews atm
(3x hackernews now lfg)
(3x hackernews now lfg)
my latest blog post "auto-research with codex: how I achieved a 212x faster kernel over baseline with codex in GPU Mode's qr_v2 problem" is up now. in this post, i talk about my approach towards auto-kerneling on the QR decomposition problem.
sankalp.bearblog.dev/autoresearch/
25
34
1,038
108,629
GPU MODE retweeted
Today, our CTO @ChrisKitching17 joins @GPU_MODE - one of the sharper corners of the accelerated HPC community - for a live talk on YouTube.
📅 7-8:30PM London / 11AM-12:30PM SF
Streamed via GPU MODE's Discord. If you spend your day in kernels, compilers, or wrestling GPUs or, come watch.
📅 7-8:30PM London / 11AM-12:30PM SF
Streamed via GPU MODE's Discord. If you spend your day in kernels, compilers, or wrestling GPUs or, come watch.
1
2
8
601
GPU MODE retweeted
(1/9) I'm thrilled to share the open-source release of Mixture-of-Kittens (MoK), our MoE megakernel for NVL72s! MoK fuses all mixture-of-experts communication and computation into a single, fully deterministic kernel, and powers Composer training across tens of thousands of GPUs.
Joint work with @nash_c_brown, @hmwildermuth, @tmwilliamlin168, and @ellev3n11
Joint work with @nash_c_brown, @hmwildermuth, @tmwilliamlin168, and @ellev3n11
We're open-sourcing Mixture-of-Kittens (MoK), our MoE training megakernel for NVL72s.
It fuses all Mixture-of-Experts communication and computation into a single, fully deterministic kernel, and runs up to 2.37x faster than the strongest public baselines.
18
45
441
47,730
@EliaIsPosting Beautiful post! Please come give us a talk! (In person if you happen to be in SF too)
1
0
4
535
The most important systems projects are all open
I stand on the shoulders of giants who stood on the shoulders of giants. So much of the progress we’ve made has been because of the openness of our great predecessors. Proud to support microsoft.com/en-us/corporat…
0
0
19
4,132
GPU MODE retweeted
If you'd like to know more, here's a nice talk at @GPU_MODE introducing the One Layer Deeper competition.🙂
youtube.com/watch?v=ustDJb…
youtube.com/watch?v=ustDJb…
Played around with this a bit. It’s nice to see folks thinking about it, and it is certainly an important problem to address.
It's a deceptively good benchmark that asks whether gradient-based learning can learn a procedure for reusable computation that generalises to more steps than it saw during training, or whether it just learns a map between inputs and outputs.
The question is whether the model can learn the serial computation itself, rather than relying primarily on test-time compute (in token space), and whether latent depth can provide a cheaper form of sequential computation. You can see the thought that has gone into designing the tasks.
Kudos to folks @coreauto and @tilderesearch.
0
2
4
1,671
We'll be doing a Q&A and demo tomorrow at 3:00pm Tuesday July 21 on our youtube channel. Please bring your questions!
Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable?
@coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper"
blem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases.
This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way.
The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank
There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation.
Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!
0
0
7
3,254
GPU MODE retweeted
Great results thus far from the 9 participants in the @GPU_MODE competition. 11B tokens and $500 in @modal credits thus far.
Thanks to @damngamerz and everyone else for participating!
Thanks to @damngamerz and everyone else for participating!
I'll be leading a sprint on the @GPU_MODE Cholesky competition at @europython / @EuroSciPy this Sat/Sun in Krakow! Come learn how to autoresearch and write CUDA kernels.
@modal has generously donated GPU access and @nvidia will provide frontier LLM access for the sprint.
6
6
49
7,511
GPU MODE retweeted
Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable?
@coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper"
blem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases.
This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way.
The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank
There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation.
Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!
@coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper"
blem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases.
This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way.
The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank
There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation.
Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!
14
57
494
78,469
GPU MODE retweeted
Second problem is now out: dense symmetric eigenproblem A=QΛQT. Solution due on July 15!
We've also enabled ncu profiling for your agents on a @verdacloud cloud box sponsored by our good friends at Brev at @NVIDIAAI
We've also enabled ncu profiling for your agents on a @verdacloud cloud box sponsored by our good friends at Brev at @NVIDIAAI
Launching a new kernel competition: Linear Algebra Kernels For The Age Of Research.
First problem: batched QR decomposition on B200. Old math, modern hardware.
Prize: Rare swag and hangout in SF
6
15
155
86,138
GPU MODE retweeted
We taught a brand-new mini-series this year at @SCSatCMU on Modern GPU Programming for ML Systems, as part of the ML Systems course, touching on fun questions like what data layout swizzling is, how to use 3D TMA, and state-of-the-art Blackwell programming. We released a curated e materials: mlc.ai/modern-gpu-pro… check it out
24
289
1,826
205,526