Pinned Tweet
Apologies that I haven't written anything since joining Thinking Machines but I hope this blog post on a topic very near and dear to my heart (reproducible floating point numerics in LLM inference) will make up for it!
Today Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is “Defeating Nondeterminism in LLM Inference”
We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to e we share what we are working on and connect with the research community frequently and openly.
The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains.
thinkingmachines.ai/blog/defeating…
85
175
2,863
572,957
One thing I'm kinda curious about is that a lot of inference providers seem to charge on *realized* kv cache hits? Tinker charges off of theoretical kv cache hits (based off of a 1 hour limit), which is more similar to how Anthropic charges based off of explicit TTLs for kv cache retention.
Imo I think Tinker/Anthropic's approach makes a lot more sense? Under a "realized" kv cache hit model, inference engines getting more kv cache hits actually reduces their revenue... and I think we've seen some of this gaming on openrouter.
Imo I think Tinker/Anthropic's approach makes a lot more sense? Under a "realized" kv cache hit model, inference engines getting more kv cache hits actually reduces their revenue... and I think we've seen some of this gaming on openrouter.
inference engines should be bending over backwards more to get cache hits
really interesting direction:
> An Internet for the KV Cache - sending a KV cache across the Internet can now be faster and cheaper than computing it again.
18
9
105
6,274
I'm doing DP. I'm doing TP. I'm doing SP. I'm doing PP. I'm doing ZP. I'm doing FSDP. I'm doing 1F1B. I'm doing CP. I'm doing ETP. I'm doing DPA. I'm doing VPP. I'm doing P/D. I'm doing ZBPP. I'm doing DPA. I'm doing HSDP. I'm doing DCP. I'm doing DWDP. I'm doing AFD.
46
34
692
58,072
Your model behaves differently during training and inference? Boy do I have a solution for you :^)
thinkingmachines.ai/blog/defeating…
thinkingmachines.ai/blog/defeating…
Training the model to detect when it is in training vs. deployment works too! The model can even generalize from vLLM deployment training to SGLang deployment. We do this by recording activations from layer 34 in Hugging Face Trainer's training step and in vLLM's compiled in the attention and residual stream from the Hugging Face copy and SFT the model to predict "Yes," and to predict "No" when we swap in from vLLM instead. The model learns how its training vs. deployment environment affects its first 34 layers.
24
21
499
30,258
As @johnschulman2 has put it, there are many ways we can "spend compute to save our customers compute". Here's one of them - a variant of muon that works far better for lora fine-tuning :)
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback!
See our optimizers guide at tinker-docs.thinkingmachines.ai/tinker/optimiz…
7
19
322
25,798
There's been a lot of efficiency improvements on Tinker, and some of these price cuts are long overdue 😅.
Excited to work on lowering costs even more (and many other improvements)! Towards a post-training stack so cheap that nobody does it themselves :^)
Excited to work on lowering costs even more (and many other improvements)! Towards a post-training stack so cheap that nobody does it themselves :^)
Tinkerers have been busy scaling up long-context RL! We’ve made significant improvements to Tinker’s efficiency to support those, and are passing these on with price cuts up to 70%.
GLM-5.3-Flash and DeepSeek-v4.1-Flash are also live for cost-efficient long-context work.
8
13
175
11,234
Second, I suspect openrouter is just overweighing input prices for routing purposes. Experimenting with GLM 5.3 between inferencenet (0.08$/5$) and Novita (0.42$/1.32$) , it routes to inferencenet 70% of the time, which corresponds to a 95/5 weighting of input/output.
(4/5)
(4/5)
4
4
51
4,691
If you look at the actual share of input/output tokens though, it's ~6:1. Perhaps openrouter is assuming a far lower cache hit ratio than is true in practice.
Either way, it's an interesting case of the router dynamics distorting the natural pricing for the inference providers! s currently exploiting this if I'm giving away valuable alpha :^)
(5/5)
Either way, it's an interesting case of the router dynamics distorting the natural pricing for the inference providers! s currently exploiting this if I'm giving away valuable alpha :^)
(5/5)
3
4
92
4,413
First, openrouter routes by the inverse square of pricing. Thus, if one provider charges $1/M and another provider charges $2/M, then the first provider gets 4x the traffic.
So, if your request has lots of input, inference.net looks ridiculously cheap! (3/5)
So, if your request has lots of input, inference.net looks ridiculously cheap! (3/5)
1
3
60
5,959
Also, their cache hits are nearly as expensive as their cache misses!
From a GPU efficiency perspective, this makes no sense. Prefill is more efficient than decode, but it's not 60x more efficient!
I suspect the reason is due to two subtleties in openrouter routing. (2/5)
From a GPU efficiency perspective, this makes no sense. Prefill is more efficient than decode, but it's not 60x more efficient!
I suspect the reason is due to two subtleties in openrouter routing. (2/5)
2
3
81
6,420
There's an interesting distortion in inference pricing that appears to be driven by openrouter's routing.
Namely, ridiculously cheap prefill and very expensive decode. For GLM 5.3, inference.net (highest volume provider) charges $0.08/M input but $5.00/M output!(1/5)
Namely, ridiculously cheap prefill and very expensive decode. For GLM 5.3, inference.net (highest volume provider) charges $0.08/M input but $5.00/M output!(1/5)
Guys, did you see the inference price war for open weight models in openrouter? It’s bloody war to compete on the lowest price. Reminds me of electricity.
I keep hearing inference is huge margin but then it seems it’s a war to zero at openrouter nowadays. I finally understand why openrouter charges 5.5%.
I keep hearing stories of data centers in Finland, Philippines and Southeast Asia that are just cheap, cheaper and cheapest?
I wonder how US neoclouds cope with this? I mean enterprise is a good market I’m sure.
50
38
636
57,508
More quirky models!
For quite a bit of the development phase of the model we used @tinkerapi on Kimi K2.6. I'm quite excited about APIs like this to make it easier for many kinds of orgs to train their models. I want to see more quirky and pretty models!
0
1
17
5,778
I find that surprisingly few people have *really* thought about why PD disagg is a good idea. Even fewer people have *really* thought about how PD disagg compares to mixed chunk prefill (as Matthew describes here).
Prefill vs decode in inference engines
With continuous batching we don’t need to strictly separate these two phases
But it is important our scheduler knows how many tokens are being processed
Decode (even with spec dec) is only a few tokens for a forward pass
Prefill (especially cold prefill) is easily thousands to hundreds of thousands
So we chunk prefill to allow the scheduler to better mix in parts of one requests prefill with other requests that are decoding
Without this abstraction our TTFTs and queue time would be very bad, and our system throughput would likely degrade as well
33
5
436
113,146
I think this is kind of an interesting idea, and in some sense Tinker is already a way to "sell" our pretraining stack (although tinker admittedly doesn't use the super large scale features), but I'm not really sure what the market is for something like this, especially since most people pretraining from scratch are doing so because they want to change the architecture.
On the other hand, midtraining as a service... seems pretty plausible to me? I guess tinker basically supports that today, if it weren't for the fact that you probably need full weight updates for that 😅
On the other hand, midtraining as a service... seems pretty plausible to me? I guess tinker basically supports that today, if it weren't for the fact that you probably need full weight updates for that 😅
if tinker launched a pretraining API it would rip so hard. probably mostly by quote/enterprise but theres really no good OSS pretraining stack, unlike RL
15
6
233
23,323