Luke Metz retweeted
#ICLR2026 Test of Time Award Talk on
"Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks"
"Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks"
3
30
275
33,322
Luke Metz retweeted
I built GoodAIList.com to help me stay-up-date with new AI stuff.
It's tracking 14K open source repos so far, with contributions from over 145K developers.
Every day, it:
- searches for new AI repos (based on 123 keywords and topics)
- surfaces repos that are gaining izes each repo
The annotations are done by AI so they are not super accurate, but they've helped me find some useful stuff.
It also lets me see where the contributors are, so when I travel, I can find folks doing cool stuff in a new city or country.
It's tracking 14K open source repos so far, with contributions from over 145K developers.
Every day, it:
- searches for new AI repos (based on 123 keywords and topics)
- surfaces repos that are gaining izes each repo
The annotations are done by AI so they are not super accurate, but they've helped me find some useful stuff.
It also lets me see where the contributors are, so when I travel, I can find folks doing cool stuff in a new city or country.
17
46
259
20,843
Luke Metz retweeted
Today, we’re launching a Rather Large™ update to the OpenAI Agents SDK.
Agents SDK now allows you to scale Codex-style agents in production, without building the whole harness yourself. We’ve brought all of the stuff of modern agents: computer-use, skills, memory, compaction, and more to the same platforms you’re already using.
Agents SDK now allows you to scale Codex-style agents in production, without building the whole harness yourself. We’ve brought all of the stuff of modern agents: computer-use, skills, memory, compaction, and more to the same platforms you’re already using.
37
94
1,310
128,163
Luke Metz retweeted
Adaption has raised $50M to build adaptive AI systems that evolve in real time.
Everything intelligent adapts. So should AI.
Everything intelligent adapts. So should AI.
194
160
1,679
202,816
Luke Metz retweeted
LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA.
m
m
83
553
3,467
1,459,501
Luke Metz retweeted
Efficient training of neural networks is difficult. Our second Connectionism post introduces Modular Manifolds, a theoretical step toward more stable and performant training by co-designing neural net optimizers with manifold constraints on weight matrices. 23
We explore a fundamental understanding of the geometry of neural network optimization.
We explore a fundamental understanding of the geometry of neural network optimization.
110
426
2,878
1,545,382
Luke Metz retweeted
Today Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is “Defeating Nondeterminism in LLM Inference”
We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to e we share what we are working on and connect with the research community frequently and openly.
The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains.
thinkingmachines.ai/blog/defeating…
We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to e we share what we are working on and connect with the research community frequently and openly.
The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains.
thinkingmachines.ai/blog/defeating…
230
1,236
7,567
3,541,731
Luke Metz retweeted
Thinking Machines’ @Luke_Metz giving a clear, beautifully simple, but extremely clever and informative tutorial on post training at @Khipu_AI — the data is likely the most important factor!
4
4
60
8,786
Super excited to share what we've been working on. It's a privilege to be working with such an amazing team!
Today, we are excited to announce Thinking Machines Lab (thinkingmachines.ai), an artificial intelligence research and product company. We are scientists, engineers, and builders behind some of the most widely used AI products and libraries, including ChatGPT, Character.ai, PyTorch, and Mistral. Our mission is to make artificial intelligence work for you by building a future where everyone has access to the knowledge and tools to make AI serve their unique needs.
We are committed to open science through publications and code releases, while focusing on human-AI collaboration that serves diverse domains. Our approach embraces co-design of research and products to enable learning from real-world deployment and rapid iteration. This work requires three core foundations: state-of-the-art model intelligence, high-quality infrastructure, and advanced multimodal capabilities. We are committed to building models at the frontier of capabilities to deliver on this promise.
If you’re interested in joining our team, consider applying here: 6wajk07p.paperform.co
9
8
242
48,278
Luke Metz retweeted
It’s done! 150,000 words, 200+ illustrations, 250 footnotes, and over 1200 reference links.
My editor just told me the manuscript has been sent to the printers.
- The ebook will be coming out later this week.
- Paperback copies should be available in a few weeks (hopefully ear). Preorder: amzn.to/49j1cGS
- The full manuscript is also accessible on O'Reilly platform: oreillymedia.pxf.io/c/5719111/2146…
This wouldn’t have been possible without the help of so many people who reviewed the early drafts, answered my thousands of questions, introduced me to fascinating use cases, or helped me see the beauty of overlooked techniques.
Thank you everyone for making this happen!
My editor just told me the manuscript has been sent to the printers.
- The ebook will be coming out later this week.
- Paperback copies should be available in a few weeks (hopefully ear). Preorder: amzn.to/49j1cGS
- The full manuscript is also accessible on O'Reilly platform: oreillymedia.pxf.io/c/5719111/2146…
This wouldn’t have been possible without the help of so many people who reviewed the early drafts, answered my thousands of questions, introduced me to fascinating use cases, or helped me see the beauty of overlooked techniques.
Thank you everyone for making this happen!
170
568
5,664
356,291
I'm leaving OpenAI after over 2 years of wild ride.
Alongside @barret_zoph , @LiamFedus , @johnschulman2 , and many others I got to build a “low key research preview” product that became ChatGPT. While we were all excited to work on it, none of us expected it to be where it is today, 100s of millions of users in a historically short amount of time. It was truly a privilege to witness its growth.
I learned so much through the process. Thank you so much @sama , @gdb , @miramurati , @ilyasut, @bobmcgrewai for giving us a chance. OpenAI is a special place.
Now, onto new things!
Alongside @barret_zoph , @LiamFedus , @johnschulman2 , and many others I got to build a “low key research preview” product that became ChatGPT. While we were all excited to work on it, none of us expected it to be where it is today, 100s of millions of users in a historically short amount of time. It was truly a privilege to witness its growth.
I learned so much through the process. Thank you so much @sama , @gdb , @miramurati , @ilyasut, @bobmcgrewai for giving us a chance. OpenAI is a special place.
Now, onto new things!
119
120
2,914
1,042,964
It has been a pleasure working with you John. I am extremely sad to see you go. Best of luck with your new adventures!
I shared the following note with my OpenAI colleagues today:
I've made the difficult decision to leave OpenAI. This choice stems from my desire to deepen my focus on AI alignment, and to start a new chapter of my career where I can return to hands-on technical work. I've decided to pursue this goal at Anthropic, where I believe I can gain new perspectives and do research alongside people deeply engaged with the topics I'm most interested in. To be clear, I'm not leaving due to lack of support for alignment research at OpenAI. On the contrary, company leaders have been very committed to investing in this area. My decision is a personal one, based on how I want to focus my efforts in the next phase of my career.
I joined OpenAI almost 9 years ago as part of the founding team after grad school. It's the first and only company where I've ever worked, other than an internship. It's also been quite a lot of fun. I'm grateful to Sam and Greg for recruiting me back at the beginning, and Mira and Bob for putting a lot of faith in me, bringing great opportunities and helping me successfully navigate various challenges. I'm proud of what we've all achieved together at OpenAI; building an unusual and unprecedented company with a public benefit mission.
I am confident that OpenAI and the teams I was part of will continue to thrive without me. Post-training is in good hands and has a deep bench of amazing talent. I get too much credit for ChatGPT -- Barret has done an incredible job building the team into the incredibly competent operation it is now, with Liam, Luke, and others. I've been heartened to see the alignment team coming together with some promising projects. With leadership from Mia, Boaz and others, I believe the team is in very capable hands.
I'm incredibly grateful for the opportunity to participate in such an important part of history and I'm proud of what we've achieved together. I'll still be rooting for you all, even while working elsewhere.
1
0
51
56,923
Luke Metz retweeted
A challenge of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt?
Predictive human preference aims to predict which model users might prefer for a specific query.
One use case is model advance that for a prompt, users will prefer Claude Instant’s response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant.
Model routing has the potential to increase response quality while reducing costs and latency.
Predictive human preference aims to predict which model users might prefer for a specific query.
One use case is model advance that for a prompt, users will prefer Claude Instant’s response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant.
Model routing has the potential to increase response quality while reducing costs and latency.
9
53
380
60,258
Luke Metz retweeted
Have you ever done a dense grid search over neural network hyperparameters? Like a *really dense* grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.
295
2,151
11,168
1,771,287
New gradient estimation technique lead by the fantastic @OscarLi101!
It provides low variance estimates gradients of unrolled, or iterative computation graphs such as those found in rl, learned optimizers, meta optimization.
If you’re at NeuRIPS go check out the poster!
It provides low variance estimates gradients of unrolled, or iterative computation graphs such as those found in rl, learned optimizers, meta optimization.
If you’re at NeuRIPS go check out the poster!
Introducing: Noise-Reuse Evolution Strategies, an unbiased, online, memory efficient, variance-reduced gradient estimator that can outperform many other methods (including Backprop) on some particularly challenging unrolled computation graph problems.
A Thread🧵
0
0
17
10,178
Luke Metz retweeted
📝Quiz time: when you have an unrolled computation graph (see figure below), how would you compute the unrolling parameters' gradients?
If your answer only contains Backprop, now it’s time to add a new method to your gradient estimation toolbox!
If your answer only contains Backprop, now it’s time to add a new method to your gradient estimation toolbox!
1
13
128
35,538
