Jeffrey Morgan retweeted
This week’s Paper Club is all about inference.
For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade.
But inference is much more than calling model.forward. through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality.
We brought together researchers and builders from @togethercompute, @ollama, @luminal_ai, and @sgl_project & @radixark to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and more efficient.
00:00 - Why inference matters more than training
08:24 - Three kinds of inference innovation
12:14 - Dan Fu (@realDanFu): the lifecycle of an inference request
42:18 - Jeff Morgan (@jmorgan): not all open model tokens are the same
1:04:31 - Joe Fioti (@joefioti): compiling models with search, not heuristics
1:30:30 - Liangsheng Yin (@lsyincs): SGLang and memory for the agent era
For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade.
But inference is much more than calling model.forward. through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality.
We brought together researchers and builders from @togethercompute, @ollama, @luminal_ai, and @sgl_project & @radixark to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and more efficient.
00:00 - Why inference matters more than training
08:24 - Three kinds of inference innovation
12:14 - Dan Fu (@realDanFu): the lifecycle of an inference request
42:18 - Jeff Morgan (@jmorgan): not all open model tokens are the same
1:04:31 - Joe Fioti (@joefioti): compiling models with search, not heuristics
1:30:30 - Liangsheng Yin (@lsyincs): SGLang and memory for the agent era
34
45
253
40,531
Jeffrey Morgan retweeted
Defensive work shouldn’t mean sending sensitive data somewhere you don’t control. You can run open models locally with Ollama, or on Ollama’s cloud with zero data retention.
More security teams are coming to @ollama to run open models for defensive work.
Defending a system means understanding how attackers get in, which means using models that will help with things like exploit analysis and red teaming. Most closed model providers only allow that through special access programs. Open models can be used by any team, on their own machines or on servers they control.
Open models can be misused and teams should run them with care. But attackers already have them, so defenders should too.
33
13
217
14,383
Jeffrey Morgan retweeted
New open models ship fast. Ollama's developers expect them on day one.
@ollama runs that workload on NVIDIA Blackwell GPUs through Crusoe Managed Inference, in Crusoe Spark™ modular data centers in Nevada.
Signed deal to production: two weeks 👇 crusoe.ai/resources/cust…
@ollama runs that workload on NVIDIA Blackwell GPUs through Crusoe Managed Inference, in Crusoe Spark™ modular data centers in Nevada.
Signed deal to production: two weeks 👇 crusoe.ai/resources/cust…
2
4
26
5,083
State of the art on-device coding. Lightning fast on Apple Silicon and RTX/DGX spark with @ollama.
Meet Flint from @Recursive_SI! A ~40B parameter MoE open model for fast, accurate agentic coding use case made to run on-device.
Flint can generate >125 tps on a M4 Pro Macbook Pro and >150 tps on a 64GB NVIDIA RTX Spark on short contexts.
Try it on Ollama today:
ollama run flint
Claude Code:
ollama launch claude --model flint
0
0
10
816
Jeffrey Morgan retweeted
We've spent the last several months scaling our compute footprint right here in the U.S., and we got to see some of it up close at the @CrusoeAI and Redwood Materials site in Nevada. Thanks @Electron_Cowboy and @ChaseLochmiller for the tour!
Excited to keep building with Crusoe zero models and full ZDR, with top performance. We look forward to sharing more soon 👀
Excited to keep building with Crusoe zero models and full ZDR, with top performance. We look forward to sharing more soon 👀
14
13
121
13,490
Jeffrey Morgan retweeted
San Francisco showed up! 🌉
@ollama's @jmorgan and @trajectorylabs' @MichaelElabd joined Crusoe's Erwan Menard to explore a fundamental choice for AI builders: closed, open, or proprietary models?
📸 Moments from an incredible evening. Stay tuned for our next AI talk.
@ollama's @jmorgan and @trajectorylabs' @MichaelElabd joined Crusoe's Erwan Menard to explore a fundamental choice for AI builders: closed, open, or proprietary models?
📸 Moments from an incredible evening. Stay tuned for our next AI talk.
2
7
27
4,163
Jeffrey Morgan retweeted
Clef from @Cloudflare is on Ollama. Run it locally or in the cloud.
27B multimodal decision model, supports text and image input and is compatible with Jev and System One APIs.
🔒Hosted in the US and with zero data retention.
27B multimodal decision model, supports text and image input and is compatible with Jev and System One APIs.
🔒Hosted in the US and with zero data retention.
22
21
266
24,160
Jeffrey Morgan retweeted
Ornith 1.5 122B is available on Ollama! Run it all locally on your device.
ollama run ornith-1.5:122b
ollama run ornith-1.5:122b
Aloha!🌺 Meet Ornith-1.5-122B: No Tradeoff Between Strong and Fast
Stronger. Faster. Leaner. Ornith-1.5-122B delivers frontier-level capabilities with exceptional inference efficiency. With superior instruction following, stronger task decomposition, and deeper understanding , it achieves state-of-the-art performance among open-source models of comparable size, while running on a single NVIDIA DGX card!!
Open weights under the MIT license.
Coding
✅SWE-bench Verified: 0.845
✅TB2.1: 0.813
✅CyberGym: 0.800
✅NL2Repo: 0.512
Agentic
✅HLE: 0.446
✅BrowseComp: 0.785
✅MCP-Atlas: 0.746
✅Toolathlon: 0.708
🤗Hugging Face: huggingface.co/ornith-ai/Orni…
We can’t wait to see what you build with Ornith-1.5-122B.
38
36
395
30,594
Jeffrey Morgan retweeted
For the next 3 days, in collaboration with @MiniMax_AI, MiniMax M3.1 Flash will be free on Ollama's Cloud!
Hosted in the US and with zero data retention.
Hosted in the US and with zero data retention.
MiniMax M3.1 Flash is on Ollama!
🔒 Hosted in the US and with zero data retention.
Fast coding and agentic workflows, with image understanding and a 1M-token context window.
Try it on Ollama Cloud:
ollama run minimax-m3.1-flash:cloud
Claude Code:
ollama launch claude --model minimax-m3.1-flash:cloud
Codex:
ollama launch codex -–model minimax-m3.1-flash:cloud
43
40
767
49,048
Jeffrey Morgan retweeted
MiniMax M3.1 Flash is on Ollama!
🔒 Hosted in the US and with zero data retention.
Fast coding and agentic workflows, with image understanding and a 1M-token context window.
Try it on Ollama Cloud:
ollama run minimax-m3.1-flash:cloud
Claude Code:
ollama launch claude --model minimax-m3.1-flash:cloud
Codex:
ollama launch codex -–model minimax-m3.1-flash:cloud
🔒 Hosted in the US and with zero data retention.
Fast coding and agentic workflows, with image understanding and a 1M-token context window.
Try it on Ollama Cloud:
ollama run minimax-m3.1-flash:cloud
Claude Code:
ollama launch claude --model minimax-m3.1-flash:cloud
Codex:
ollama launch codex -–model minimax-m3.1-flash:cloud
MiniMax M3.1 Flash: Space Performance + Bunny Speed
🚀🐰 Yes, Space Bunny - which has topped global leaderboards with 200T+ tokens - was an earlier version of M3.1 Flash.
💰Redefining the performance economics of agentic AI. M3.1 Flash achieves frontier-level intelligence s Cowork (Automation-Bench: 62%; ALE: 30%), Coding (DeepSWE: 74%) and Visual Reasoning (BabyVision: 91%), at 1/3 the cost of M3 and 3x the speed.
API pricing (flat rate): $0.15 / $0.60 per 1M input/output tokens, with cache reads at $0.015 / 1M tokens.
Model weights will arrive on Oct 23.
API: platform.minimax.io
M Plan: minimax.io/m-plan
MiniMax Code for desktop: agent.minimax.io/download
48
29
414
71,601
Jeffrey Morgan retweeted
✈ DeepSeek-V4.1-Flash-1017 is on Ollama! US-based hosting with zero data retention.
It's strong on agentic tasks while being cheaper per task.
Try it on Ollama Cloud:
ollama run deepseek-v4.1-flash:1017-cloud
Claude Code:
ollama launch claude --model deepseek-v4.1-flash:1017-cloud
OpenCode:
ollama launch opencode --model deepseek-v4.1-flash:1017-cloud
It's strong on agentic tasks while being cheaper per task.
Try it on Ollama Cloud:
ollama run deepseek-v4.1-flash:1017-cloud
Claude Code:
ollama launch claude --model deepseek-v4.1-flash:1017-cloud
OpenCode:
ollama launch opencode --model deepseek-v4.1-flash:1017-cloud
🚀 DeepSeek-V4.1-Flash-1017 is here!
Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API.
🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, grade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model.
1/4
108
92
1,323
148,254
Jeffrey Morgan retweeted
.@ollama is now an inference provider on Replicas. Run open models in Claude Code, Codex, OpenCode and Pi
10
5
36
6,782
Jeffrey Morgan retweeted
Working on getting our US hosts up and running. Very soon!
🚀 DeepSeek-V4.1-Flash-1017 is here!
Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API.
🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, grade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model.
1/4
38
23
597
24,368
Jeffrey Morgan retweeted
In this video, Joseph asks Delta to configure itself to use @ollama as a custom OpenAI-compatible provider. It all happens in the thread, without Joseph ever touching Settings.
9
6
147
17,233
Jeffrey Morgan retweeted
안녕하세요 🇰🇷
Bags packed. Ollama is headed to Seoul ✈️
Korea is one of the most innovative AI markets in the world, and with such an active Ollama community in Seoul, we’ve been wanting to make this trip for a while. We’re really excited to finally go.
This November, we will s, customers and partners, learning how we can further grow in the market.
We’re also teaming up with @friendliai while we’re in Seoul. More on that to follow 👀
Bags packed. Ollama is headed to Seoul ✈️
Korea is one of the most innovative AI markets in the world, and with such an active Ollama community in Seoul, we’ve been wanting to make this trip for a while. We’re really excited to finally go.
This November, we will s, customers and partners, learning how we can further grow in the market.
We’re also teaming up with @friendliai while we’re in Seoul. More on that to follow 👀
67
110
595
30,388
Jeffrey Morgan retweeted
We’re excited to be day 0 partners for @reflection_ai’s first open-weight model! For a limited time,
Beam is FREE on Ollama! In partnership with @reflection_ai this model is hosted in the US with full ZDR.
It has a 1M token context window and is designed for software engineering, reasoning, and long-running agentic tasks.
Run Beam on Ollama’s Cloud:
ollama run beam
Claude Code
ollama launch claude --model beam:cloud
Codex
ollama launch codex --model beam:cloud
OpenCode
ollama launch opencode --model beam:cloud
Free to use for a limited time only.
Beam is FREE on Ollama! In partnership with @reflection_ai this model is hosted in the US with full ZDR.
It has a 1M token context window and is designed for software engineering, reasoning, and long-running agentic tasks.
Run Beam on Ollama’s Cloud:
ollama run beam
Claude Code
ollama launch claude --model beam:cloud
Codex
ollama launch codex --model beam:cloud
OpenCode
ollama launch opencode --model beam:cloud
Free to use for a limited time only.
Beam is now available across platforms, with open weights under the Apache 2.0 license - in collaboration with an ecosystem of 40+ partners.
We're releasing:
- Weights, tokenizer, and reference implementation
- BF16, FP8 and NVFP4 weights and a DFlash2 drafter
- The open source ained Beam on: Mirror CLI
Weights and model card on Hugging Face: huggingface.co/reflection
48
30
309
108,159