Pinned Tweet
For the last few years, a lot of my work has been driven by the feeling that deep learning is not magic — there are principles, mechanisms, and laws waiting to be understood.
This paper is our attempt to say that clearly!
This paper is our attempt to say that clearly!
1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics!
We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics.
🔨 arxiv.org/pdf/2604.21691 🔧
7
13
77
9,971
Had a great time visiting @TTIC_Connect last year -- the ideas I presented have progressed quite a bit since then:
- arxiv.org/pdf/2506.06489
- arxiv.org/pdf/2602.03655
- arxiv.org/pdf/2506.06489
- arxiv.org/pdf/2602.03655
1/4 The Dynamics of Deep Learning. "Trying to understand deep learning...comes down to understanding this interaction of three components that make the deep learning system," said @KuninDaniel, speaking at TTIC in April last year. Video: youtube.com/watch?v=4eMW8H…
0
5
40
4,816
The Miller Fellowship has been an awesome opportunity and Berkeley is a pretty great place to be!
Scientist @KuninDaniel (MF 25-28) studies learning in neural networks to uncover principles of intelligence in artificial & biological systems. The Miller Fellowship gives me freedom to take risks and grow as a researcher & communicator. Nominate by 9/10: bit.ly/millerfellowsh…
0
1
27
3,200
Daniel Kunin retweeted
If you’re a researcher in math or CS wondering what will be left to work on as AI charges through our fields, rest assured: the final theory is still out there waiting for us. It’s not even fully formulated let alone solved. What is the structure and form of thought itself?
4
42
378
53,746
New work by @_AmilDravid, extending his really neat work on Rosetta Neurons through the lens of scaling laws!
A nice example of how mech. interp and learning mechanics can complement one another, bringing different perspectives that together lead to deeper insights
A nice example of how mech. interp and learning mechanics can complement one another, bringing different perspectives that together lead to deeper insights
Scaling laws describe how loss changes with scale. Do neurons inside models change predictably too?
We study vision and language models up to 30B params and find systematic scaling in neuron universality, specialization, and selectivity.
Paper+code: avdravid.github.io/rosetta-neuron…
1/n
0
1
8
1,169
Daniel Kunin retweeted
A longstanding dream of interp is to decompose activations into distinct, interpretable parts.
But when should we expect that to work, and what even are such parts?
New from Simplex: transformers factor their world into orthogonal subspaces, even when it costs accuracy.🧵👇
But when should we expect that to work, and what even are such parts?
New from Simplex: transformers factor their world into orthogonal subspaces, even when it costs accuracy.🧵👇
16
98
620
82,864
Excited to share that our paper “Sequential Group Composition: A Window into the Mechanics of Deep Learning” was accepted to ICML 2026 in Seoul!
Co-led with @giovannimarchet
and @AdeleMyersPhD @hopfbifurcator @ninamiolane
Paper: arxiv.org/abs/2602.03655
Co-led with @giovannimarchet
and @AdeleMyersPhD @hopfbifurcator @ninamiolane
Paper: arxiv.org/abs/2602.03655
8
48
235
73,227
Yes — by leveraging associativity.
We explicitly construct efficient solutions: RNNs can compose elements sequentially in k steps, while deep MLPs can compose adjacent pairs in parallel in log k layers
and we find preliminary evidence that GD can discover these solutions!
We explicitly construct efficient solutions: RNNs can compose elements sequentially in k steps, while deep MLPs can compose adjacent pairs in parallel in log k layers
and we find preliminary evidence that GD can discover these solutions!
1
3
32
942
For me, this paper is learning mechanics in action!
Mech interp first identified that NNs use Fourier features in algebraic tasks - great work @bilalchughtai_ @justanotherlaw @NeelNanda5
Learn mech asks why training produced those features, in that order, with that architecture
Mech interp first identified that NNs use Fourier features in algebraic tasks - great work @bilalchughtai_ @justanotherlaw @NeelNanda5
Learn mech asks why training produced those features, in that order, with that architecture
0
1
26
696
Daniel Kunin retweeted
From "Mathematical theory of deep learning: Can we do it? Should we do it?" to "There Will Be a Scientific Theory of Deep Learning".
It's respectively the title of a talk I gave four years ago, and the title of an arxiv paper from four days ago.
I really like the "learning mechanics" perspective (think of it as a continuation of "statistical mechanics", "quantum mechanics", and so on). Several of my last academic papers can be viewed under that lens (e.g. Learning threshold neurons via the “edge of stability”; or LEGO). I'm not as optimistic as the authors of the recent arxiv paper that we will EVER be able to reach what the "physics mechanics" field have achieved, but it's certainly worth trying.
Talk: youtu.be/3uRD_lg701k?si…
Paper: arxiv.org/abs/2604.21691
It's respectively the title of a talk I gave four years ago, and the title of an arxiv paper from four days ago.
I really like the "learning mechanics" perspective (think of it as a continuation of "statistical mechanics", "quantum mechanics", and so on). Several of my last academic papers can be viewed under that lens (e.g. Learning threshold neurons via the “edge of stability”; or LEGO). I'm not as optimistic as the authors of the recent arxiv paper that we will EVER be able to reach what the "physics mechanics" field have achieved, but it's certainly worth trying.
Talk: youtu.be/3uRD_lg701k?si…
Paper: arxiv.org/abs/2604.21691
17
59
416
53,583
Daniel Kunin retweeted
Wir mussen wissen. Wir werden wissen
There Will Be a Scientific Theory of Deep Learning
Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix-Adserà, Blake Bordelon, Jeremy Cohen, Nikhil Ghosh, Florentin Guth, Arthur Jacot, Mason Kamb, Dhruva Karkada, …
arxiv.org/abs/2604.21691 [𝚜𝚝𝚊𝚝.𝙼𝙻 𝚌𝚜.𝙻𝙶]
1
2
21
3,475
Daniel Kunin retweeted
Very happy to be part of this project. We've compiled the main reasons why a Theory of Deep Learning is possible if not inevitable!
1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics!
We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics.
🔨 arxiv.org/pdf/2604.21691 🔧
0
1
18
2,142
Daniel Kunin retweeted
Great perspective on the theory of deep learning from a stellar group of authors!Physics-inspired ideas will play a central role in shaping this field. Congrats to my group alumni @blake__bordelon and @ABAtanasov for their contributions here and across many influential papers.
1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics!
We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics.
🔨 arxiv.org/pdf/2604.21691 🔧
1
4
35
3,797
100% agree. Neuroscience embraces studying the brain at multiple levels — computational, algorithmic, and implementational. I’m excited to see deep learning moving toward the same conversation, with theory and interpretability informing each other!
It's been so heartening to see deep learning theory folks engage seriously with interpretability recently, and I hope these two communities can talk much, much more. We should seek a unified understanding of neural networks across many levels of analysis.
0
2
25
1,802
Daniel Kunin retweeted
Great to see the next generation taking the lead in the science of deep learning! Also proud that two brilliant members/alumni of my group are a part of this: @KuninDaniel & @MasonKamb
1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics!
We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics.
🔨 arxiv.org/pdf/2604.21691 🔧
2
14
98
10,867
Daniel Kunin retweeted
There Will Be a Scientific Theory of Deep Learning ift.tt/FIXLaes
4
48
310
29,007