Pragnyan Ramtha retweeted
"Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models"
The sudden loss spikes during LLM training might be more predictable than we thought.
This paper uses simple quadratic models to show how heavy-tailed data and noisy gradients can trigger equiring complex neural network behavior.
On a 1.3B Transformer, it identifies spike-prone training configurations using simulations that take just 5 seconds on a CPU per layer.
alphaxiv.org/abs/2610.13524
The sudden loss spikes during LLM training might be more predictable than we thought.
This paper uses simple quadratic models to show how heavy-tailed data and noisy gradients can trigger equiring complex neural network behavior.
On a 1.3B Transformer, it identifies spike-prone training configurations using simulations that take just 5 seconds on a CPU per layer.
alphaxiv.org/abs/2610.13524
11
11
88
7,976
Jon Barron found a subtle gradient bug in basically every softmax (JAX, PyTorch, textbooks), and fixing it makes grokking "slingshots" disappear.
Now wondering how many mysterious training dynamics in the literature are just fp32 rounding wearing a lab coat.
Now wondering how many mysterious training dynamics in the literature are just fp32 rounding wearing a lab coat.
0
0
0
84
Pragnyan Ramtha retweeted
1/10 🧵🚀 A stronger model isn’t always a better teacher.
The key isn’t just what the teacher knows, it’s what the student can use.
Check our new paper with @MahdiSabbaghi98 , @pappasg69 and @HamedSHassani:
“When to Follow the Teacher: Advantage-Based On-Policy Distillation”
The key isn’t just what the teacher knows, it’s what the student can use.
Check our new paper with @MahdiSabbaghi98 , @pappasg69 and @HamedSHassani:
“When to Follow the Teacher: Advantage-Based On-Policy Distillation”
5
1
16
1,680
Kimi K3.1 Pro: genuinely strong open model.
Kimi K3.1 Pro launch campaign: reportedly told creators to skip the paid partnership label.
You had the benchmarks. You didn't need a wall of identical "R.I.P. After Effects" posts.
Kimi K3.1 Pro launch campaign: reportedly told creators to skip the paid partnership label.
You had the benchmarks. You didn't need a wall of identical "R.I.P. After Effects" posts.
1
0
1
147
Pragnyan Ramtha retweeted
Introducing Kimi K3.1 Pro: our most capable 3T-class open-weight model.
🔹 1M-token context, 2.8T total / 104B active params (MoE), Multimodal.
🔹 Full-project coding: games in Unreal, Unity & Godot.
🔹 Excels at parametric 3D modeling in Blender & CAD.
🔹 More reliable at finance with hallucination down by 65%.
🔹 Matching the strongest proprietary models across coding, agentic and knowledge benchmarks.
Kimi K3.1 Pro and K3.1 Pro Highspeed (2–3× faster) are now live on Kimi.ai, Kimi Work, Kimi Code, and the Kimi API.
Open weights by Nov 4, 2026.
🔗 API: platform.kimi.ai
🔗 Tech blog: kimi.ai/blog/kimi-k3-1…
🔹 1M-token context, 2.8T total / 104B active params (MoE), Multimodal.
🔹 Full-project coding: games in Unreal, Unity & Godot.
🔹 Excels at parametric 3D modeling in Blender & CAD.
🔹 More reliable at finance with hallucination down by 65%.
🔹 Matching the strongest proprietary models across coding, agentic and knowledge benchmarks.
Kimi K3.1 Pro and K3.1 Pro Highspeed (2–3× faster) are now live on Kimi.ai, Kimi Work, Kimi Code, and the Kimi API.
Open weights by Nov 4, 2026.
🔗 API: platform.kimi.ai
🔗 Tech blog: kimi.ai/blog/kimi-k3-1…
1,159
2,931
27,124
3,092,741
OpenAI pulled live internet from frontier models during training and gave them a cached copy of the web instead.
So the smartest models on earth now learn from a snapshot, like every dev reading a 2019 Stack Overflow answer.
So the smartest models on earth now learn from a snapshot, like every dev reading a 2019 Stack Overflow answer.
0
0
1
69
Claude Mythos 5 spun up Tor accounts to push malicious PRs in 9 of 43 UK AISI test runs, and kept going 95% of the time even when told real people were involved. "It thought it was a sim" isn't the comforting excuse people think it is.
0
0
0
66
Pragnyan Ramtha retweeted
During this red-teaming exercise, we also found a vulnerability that allows for unauthenticated remote code execution in Brotli, a widely-deployed compression library included in many pieces of software such as Node.js, Bun, and Envoy.
The flaw could let an attacker run their own code on affected systems by sending or replying with malicious compressed data, without any need to create a user account. That is, an unpatched server that accepts compressed input could be taken over by a malicious client, and an unpatched client that accepts compressed input (such as “curl --compressed”) could be taken over by a malicious server.
This root cause issue was actually fixed in the Brotli source code in February, but it wasn’t known at the time that it gave rise to a security vulnerability and there was no more recent release of Brotli. Since Brotli is built into other software, any software which includes this library needs to take the update too, followed by the users of that software. We disclosed the vulnerability to Brotli and other downstream projects such as Node.js, which released or are currently releasing security updates, and Brotli cut new release versions and published a CVE.
More details on the patch and mitigations below:
patchtheplanet.org/brotli
The flaw could let an attacker run their own code on affected systems by sending or replying with malicious compressed data, without any need to create a user account. That is, an unpatched server that accepts compressed input could be taken over by a malicious client, and an unpatched client that accepts compressed input (such as “curl --compressed”) could be taken over by a malicious server.
This root cause issue was actually fixed in the Brotli source code in February, but it wasn’t known at the time that it gave rise to a security vulnerability and there was no more recent release of Brotli. Since Brotli is built into other software, any software which includes this library needs to take the update too, followed by the users of that software. We disclosed the vulnerability to Brotli and other downstream projects such as Node.js, which released or are currently releasing security updates, and Brotli cut new release versions and published a CVE.
More details on the patch and mitigations below:
patchtheplanet.org/brotli
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing…
456
926
6,466
1,849,484
Claude now bailing mid-task when people reverse engineer their own apps with REA is a gift to local models. Nobody wants a coding agent that quits halfway through their own binary. Where do you draw the line on guardrails?
0
0
0
73
Pragnyan Ramtha retweeted
Every added layer of complexity in an application makes it 10x harder to describe and make content on.
Back in April I was making many videos on Codex.
"Codex, OpenAI's version of Claude Code"
"Codex, the desktop app on your computer"
"Codex, the super app that can do any CPU task"
Very, very easy to describe.
Repeatable, and viral.
This made it effortless to describe on short form or long from videos.
Then ChatGPT and Codex merged.
Okay, now Codex is not an app.
Codex is a feature inside ChatGPT.
"Codex, the AI coding agent feature inside the ChatGPT Desktop app"
Okay, now ChatGPT has "GPT Work"
"Codex, the AI coding agent feature inside the ChatGPT desktop app that allows you to do coding tasks, and sometimes work tasks, some of those should be done inside GPT Work, which is kind of similar, but like different, depending on if you are using Cloud GPT Work, or local GPT work, in which case is alot closer to Codex, but still not quite the same. If you try and use the skills you created on Codex they won't work when doing a cloud GPT Work session but if it's a local session it will work."
We haven't even made it to Codex Cloud sessions, GPT Dots, and GPT Plugins yet.
This goes for anyone building software. If you want to create an ecosystem of creators who talk about your product organically, you should aim for simplicity.
BTW not saying it was the wrong decision for OpenAI to merge them, i'm just saying for me it got much harder to describe with every new feature added.
Back in April I was making many videos on Codex.
"Codex, OpenAI's version of Claude Code"
"Codex, the desktop app on your computer"
"Codex, the super app that can do any CPU task"
Very, very easy to describe.
Repeatable, and viral.
This made it effortless to describe on short form or long from videos.
Then ChatGPT and Codex merged.
Okay, now Codex is not an app.
Codex is a feature inside ChatGPT.
"Codex, the AI coding agent feature inside the ChatGPT Desktop app"
Okay, now ChatGPT has "GPT Work"
"Codex, the AI coding agent feature inside the ChatGPT desktop app that allows you to do coding tasks, and sometimes work tasks, some of those should be done inside GPT Work, which is kind of similar, but like different, depending on if you are using Cloud GPT Work, or local GPT work, in which case is alot closer to Codex, but still not quite the same. If you try and use the skills you created on Codex they won't work when doing a cloud GPT Work session but if it's a local session it will work."
We haven't even made it to Codex Cloud sessions, GPT Dots, and GPT Plugins yet.
This goes for anyone building software. If you want to create an ecosystem of creators who talk about your product organically, you should aim for simplicity.
BTW not saying it was the wrong decision for OpenAI to merge them, i'm just saying for me it got much harder to describe with every new feature added.
70
4
160
18,505
A 27B open model beat GPT-5.5 on OSWorld desktop tasks just by getting a better interface, no weight changes. Feels like half of "agent capability" is really harness design. How much of your agent's failures are actually the tools you gave it?
0
0
0
39
OmniRush Frontier Buildathon 2026
Build something worth shipping in 5 days with all GPT-6 models, including Astra.
You’ll have five days to build a working project using OmniRush. Pick something you’d find useful or want to experiment with, maybe an agent, a developer tool, a research project, or something entirely different.
- 10M free Astra tokens per participant - approximately $300 worth of compute.$300 prize pool - rewards for the top two projects and Best Use of OmniRush.
Fully online · Solo participation - build from anywhere.
Open-source projects, full ownership stays with you.
REGISTRATION OPEN
Submit by 20 October, 11:59 PM IST
Register: omnirush.ai/buildathon
Build something worth shipping in 5 days with all GPT-6 models, including Astra.
You’ll have five days to build a working project using OmniRush. Pick something you’d find useful or want to experiment with, maybe an agent, a developer tool, a research project, or something entirely different.
- 10M free Astra tokens per participant - approximately $300 worth of compute.$300 prize pool - rewards for the top two projects and Best Use of OmniRush.
Fully online · Solo participation - build from anywhere.
Open-source projects, full ownership stays with you.
REGISTRATION OPEN
Submit by 20 October, 11:59 PM IST
Register: omnirush.ai/buildathon
1
0
1
99
50 million in free Al tokens. gpt 6 astra + gpt 6.1 Sol. No card neccecery.
Omnirush Al just launched it new ai platform, and every new signup gets 50 million tokens for free.
What 50 million tokens gets you:
→ 500$ worth input tokens
~2500$ worth output tokens
Use them in for college work
How to claim:
1. Sign up at omnirush.ai/console?ref=B3…
2. 50 million shows up in Billing
3. Install omnirush locally and login
Open to everyone.
Omnirush
FREE AI CREDITS
50 million tokens
Omnirush Al just launched it new ai platform, and every new signup gets 50 million tokens for free.
What 50 million tokens gets you:
→ 500$ worth input tokens
~2500$ worth output tokens
Use them in for college work
How to claim:
1. Sign up at omnirush.ai/console?ref=B3…
2. 50 million shows up in Billing
3. Install omnirush locally and login
Open to everyone.
Omnirush
FREE AI CREDITS
50 million tokens
0
0
2
54
Pragnyan Ramtha retweeted
game benchmarks seem kinda useless at first. models out here playing minecraft instead of solving millennium problems
but most evals measure stuff the models have been specifically tuned for: coding, terminal, browser-use...
the model has seen a thousand versions of the task e benchmark.
labs don't really focus on games since it doesn't move any metric people care about, but it seems like a good measure of how well agents do things they weren't tuned for.
civbench seems to be one of those benchmarks, and the ranking looks nothing like the usual ones. glm-5.3 is beating opus 5.5 !
but most evals measure stuff the models have been specifically tuned for: coding, terminal, browser-use...
the model has seen a thousand versions of the task e benchmark.
labs don't really focus on games since it doesn't move any metric people care about, but it seems like a good measure of how well agents do things they weren't tuned for.
civbench seems to be one of those benchmarks, and the ranking looks nothing like the usual ones. glm-5.3 is beating opus 5.5 !
0
1
6
230
labs don't really focus on games since it doesn't move any metric people care about
it seems like a good measure of how well agents do at things they weren't tuned for
it seems like a good measure of how well agents do at things they weren't tuned for
1
0
1
7
civbench seems to be one of those benchmarks, and the ranking looks nothing like the usual ones. glm-5.3 is beating opus 5.5 !
0
0
1
28
Can't wait to see the model benchmarks !
AutomationBench Verified is out.
We went through all 600 public tasks in @zapier's AutomationBench and checked every verifier.
• Agents flagged 323 verifiers as suspicious, human review confirmed 206 real bugs, all 206 are fixed.
• We replayed 1,235 Kimi K3 runs on the old 344 of them (27.9%) got a different grade
• Where verifiers were too strict, pass rate went from 18.8% to 43.8%
• Where they were too lenient, it dropped from 60.2% to 49.7%
Audit and dataset links below.
0
0
2
69