Follow-up: Intel's 32GB Arc cards just got another Local AI speedup.
TensorFold 1.2.1 landed after this post, and Gemma 4 E4B prompt processing on ONE Arc Pro B70 32GB is now:
2K context: 6,778 → 15,606 tps
8K: 6,039 → 14,446 tps
32K: 4,232 → 10,573 tps
65K: 3,042 → 7,886 –2.6× faster prefill.
And TensorFold says the logits are bit-identical.
At a 52K-token chat prompt
⏱️ TTFT: 15.2 sec → 6.2 sec
⚡ decode: 92.7 → 97.9 tps
Four 52K requests at once: 62.0 sec → 25.9 sec
Link in ALT
TensorFold 1.2.1 landed after this post, and Gemma 4 E4B prompt processing on ONE Arc Pro B70 32GB is now:
2K context: 6,778 → 15,606 tps
8K: 6,039 → 14,446 tps
32K: 4,232 → 10,573 tps
65K: 3,042 → 7,886 –2.6× faster prefill.
And TensorFold says the logits are bit-identical.
At a 52K-token chat prompt
⏱️ TTFT: 15.2 sec → 6.2 sec
⚡ decode: 92.7 → 97.9 tps
Four 52K requests at once: 62.0 sec → 25.9 sec
Link in ALT
Intel's 32GB Arc cards are getting so much better for Local AI. 👀
One Arc Pro B65 32GB is now running Qwen3.8-Flash-Next at these levels
⚡ 62.4 tps
⏱️ 2.1 sec TTFT
💾 32GB VRAM
🧠 experts offloaded to system RAM
Even the default XeStrata recipe hit 57.5 tps.
And Strata 0.2.8 s even better.
2× Arc Pro B65
💬 Chat
~60–63 → 71–74 tps
📊 JSON
~67–68 → 80–83 tps
That's a reported 17–23% decode gain, with identical answers in the paired tests.
Links in ALT
2
1
3
516
DeepSeek V4.1 Flash 1017 reaching 5.8% on ARC-AGI-3 is crazy! 👀
Just acknowledging the harness is helping
🧠 Standard harness: 0.2%
🛠️ Provider Adapter harness: 5.8%
Same model and both at maximum reasoning.
The adapter preserves reasoning state between requests and compacts the model can reuse earlier work.
Just acknowledging the harness is helping
🧠 Standard harness: 0.2%
🛠️ Provider Adapter harness: 5.8%
Same model and both at maximum reasoning.
The adapter preserves reasoning state between requests and compacts the model can reuse earlier work.
4
0
7
787
@DeryaTR_ Yes confirmed. x.com/TeksEdge/statu…
Been watching this post since it was a wee 10K views.
So basically, imagine using a large AI once to build a tiny neural network you can run locally. 👀
@AlexKrentsel reports reproducing the paper's results on a weight-space programming experiment.
Given the rules and code for a simplified "Breakout" game, frontier models wrote all 20,486 parameters of a small game-playing network in a single response.
No tools or training loop to create that network.
In his reproduction, the best generated network scored 22× the experiment’s PPO-trained baseline. 👈
The Localmaxxer possibility is intriguing
🧠 A large model constructs the weights
💾 You keep the resulting tiny network
🖥️ Your hardware runs it without calling the large model for every decision
These are small, specialized experiments. Reliability on unfamiliar tasks and scaling to larger networks remain open questions.
Using powerful AI to build useful little models we can run ourselves? Sooo cool
1
0
2
274
A new local-agent harness has a built-in BS detector.
Rufous is an Apache-2.0 Rust harness built specifically for small 2B–12B local models.
This should help w/Local AI.
If the model says “I created the file.” …but its tool-call history shows it never actually created the esponse and makes the model correct itself.
It’s designed for Local AI for inference engines like
🦙 Ollama
🖥️ LM Studio
⚙️ llama-server
🔌 any OpenAI-compatible endpoint
It also starts with only ~1,250 tokens of prefill containing 9 core tools, then loads web, memory and MCP tools only when needed.
Rufous is trying to make the HARNESS smarter.
Link in ALT
Rufous is an Apache-2.0 Rust harness built specifically for small 2B–12B local models.
This should help w/Local AI.
If the model says “I created the file.” …but its tool-call history shows it never actually created the esponse and makes the model correct itself.
It’s designed for Local AI for inference engines like
🦙 Ollama
🖥️ LM Studio
⚙️ llama-server
🔌 any OpenAI-compatible endpoint
It also starts with only ~1,250 tokens of prefill containing 9 core tools, then loads web, memory and MCP tools only when needed.
Rufous is trying to make the HARNESS smarter.
Link in ALT
6
2
11
1,020
AMD Local AI users, here's a vLLM update worth watching for your next build.
Octave KV, merged into vLLM, compresses the attention cache used while a model processes and generates text.
🎯 The author reports 4.6× the cached-token capacity with Qwen3.8-27B on 4× AMD RX 7900 XTX, ache.
That could allow longer conversations or more simultaneous requests within the model’s context limit.
This compresses the KV cache, not the model weights.
The author also reports comparable results on selected math and long-context retrieval tests.
🔗 Link in ALT.
Octave KV, merged into vLLM, compresses the attention cache used while a model processes and generates text.
🎯 The author reports 4.6× the cached-token capacity with Qwen3.8-27B on 4× AMD RX 7900 XTX, ache.
That could allow longer conversations or more simultaneous requests within the model’s context limit.
This compresses the KV cache, not the model weights.
The author also reports comparable results on selected math and long-context retrieval tests.
🔗 Link in ALT.
5
0
19
1,390
@loktar00 I know and as they say “this is the worst they will ever be”
0
0
1
40
@loktar00 This is soooop great!!! I think I’ll try making a Family Fued where each person gets AI family
1
0
2
58
Been watching this post since it was a wee 10K views.
So basically, imagine using a large AI once to build a tiny neural network you can run locally. 👀
@AlexKrentsel reports reproducing the paper's results on a weight-space programming experiment.
Given the rules and code for a simplified "Breakout" game, frontier models wrote all 20,486 parameters of a small game-playing network in a single response.
No tools or training loop to create that network.
In his reproduction, the best generated network scored 22× the experiment’s PPO-trained baseline. 👈
The Localmaxxer possibility is intriguing
🧠 A large model constructs the weights
💾 You keep the resulting tiny network
🖥️ Your hardware runs it without calling the large model for every decision
These are small, specialized experiments. Reliability on unfamiliar tasks and scaling to larger networks remain open questions.
Using powerful AI to build useful little models we can run ourselves? Sooo cool
So basically, imagine using a large AI once to build a tiny neural network you can run locally. 👀
@AlexKrentsel reports reproducing the paper's results on a weight-space programming experiment.
Given the rules and code for a simplified "Breakout" game, frontier models wrote all 20,486 parameters of a small game-playing network in a single response.
No tools or training loop to create that network.
In his reproduction, the best generated network scored 22× the experiment’s PPO-trained baseline. 👈
The Localmaxxer possibility is intriguing
🧠 A large model constructs the weights
💾 You keep the resulting tiny network
🖥️ Your hardware runs it without calling the large model for every decision
These are small, specialized experiments. Reliability on unfamiliar tasks and scaling to larger networks remain open questions.
Using powerful AI to build useful little models we can run ourselves? Sooo cool
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: fronely read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weight-sp…
5
0
22
1,646
Small local models can do useful work, contrary to some.
Getting the best out of them takes intentional task design.
They often need more guidance than stronger models, especially when a task requires reasoning across many dependent steps or recalling facts accurately.
Here's er cloud model, or a larger local model, plan the work, break it into manageable tasks, and review the results.
Give the smaller local model bounded jobs (your managing AI should be told to plan this way) such as code changes, execute a clearly defined task, extract information from supplied documents, run tests, and report what happened.
Give it the relevant context, clear constraints, and a way to check success.
The Localmaxxer skill is learning how much responsibility each model can reliably handle.
Getting the best out of them takes intentional task design.
They often need more guidance than stronger models, especially when a task requires reasoning across many dependent steps or recalling facts accurately.
Here's er cloud model, or a larger local model, plan the work, break it into manageable tasks, and review the results.
Give the smaller local model bounded jobs (your managing AI should be told to plan this way) such as code changes, execute a clearly defined task, extract information from supplied documents, run tests, and report what happened.
Give it the relevant context, clear constraints, and a way to check success.
The Localmaxxer skill is learning how much responsibility each model can reliably handle.
8
5
61
5,518
It’s sad to say, but the warnings earlier this year about securing your local AI hardware are looking harder to dismiss.
A new Wccftech report says some GPU vendors are steering distributors toward older RTX 30-series and Radeon cards because RTX 50-series supply is tight.
The ate, describes shortages amid mining demand and China’s e-commerce promotional season, alongside rising VRAM and component costs.
Older cards can still do useful work. But the report suggests even their prices could remain firm or rise.
Still, owning your AI depends on being able to buy the hardware to run it.
Link to article in ALT
A new Wccftech report says some GPU vendors are steering distributors toward older RTX 30-series and Radeon cards because RTX 50-series supply is tight.
The ate, describes shortages amid mining demand and China’s e-commerce promotional season, alongside rising VRAM and component costs.
Older cards can still do useful work. But the report suggests even their prices could remain firm or rise.
Still, owning your AI depends on being able to buy the hardware to run it.
Link to article in ALT
2
1
12
3,542
Someone squeezed a Qwen3.8-27B fine-tune down to ~7.4GiB on the GPU. Not sure I believe it.
And somehow it just scored higher than Qwen3.6-27B BF16 on LiveCodeBench. 👀
The new Swift 1.5 / Qwen3.8-27B ATX AI2 build is only ~2.3 bits per weight.
Stats
💾 ~7.4GiB resident on ameters
⚡ >90 t/s reported for agent work on 3090/3080 Ti
📚 designed to make 200K+ context possible on 12GB Ampere cards
⚙️ custom llamAmpere runtime
This is almost unbelievable.
LiveCodeBench v6:
Qwen3.8 27B 2.3bpw → 84.3%
Qwen3.6 27B BF16 → 83.9%
Qwen3.8 27B BF16 → 90.3%
So no, the 2.3-bit model hasn't magically preserved all of Qwen3.8.
But a ~7.4GiB-on-GPU version is landing around the coding performance of the previous generation's full BF16 model.
And the compression quality improved substantially too.
KLD
old 2.3-bit build → 0.0963
new build → 0.0594
~38% less quantization error.
The author says the layout enables 200K+ context on 12GB cards, but the exact 12GB config hadn't yet been tested on this new checkpoint when the model card was published.
Link in ALT
And somehow it just scored higher than Qwen3.6-27B BF16 on LiveCodeBench. 👀
The new Swift 1.5 / Qwen3.8-27B ATX AI2 build is only ~2.3 bits per weight.
Stats
💾 ~7.4GiB resident on ameters
⚡ >90 t/s reported for agent work on 3090/3080 Ti
📚 designed to make 200K+ context possible on 12GB Ampere cards
⚙️ custom llamAmpere runtime
This is almost unbelievable.
LiveCodeBench v6:
Qwen3.8 27B 2.3bpw → 84.3%
Qwen3.6 27B BF16 → 83.9%
Qwen3.8 27B BF16 → 90.3%
So no, the 2.3-bit model hasn't magically preserved all of Qwen3.8.
But a ~7.4GiB-on-GPU version is landing around the coding performance of the previous generation's full BF16 model.
And the compression quality improved substantially too.
KLD
old 2.3-bit build → 0.0963
new build → 0.0594
~38% less quantization error.
The author says the layout enables 200K+ context on 12GB cards, but the exact 12GB config hadn't yet been tested on this new checkpoint when the model card was published.
Link in ALT
7
8
90
5,919
Tencent released a new research-agent harness. It's not being used as you might think.
We are entering a world of RSI.
Tencent's new harness trained a tiny 1.96M-parameter model from scratch that's 57× faster than the tool it was trying to replace.
And I think this is a ction for Local AI.
Tencent just introduced Hyra 2.0.
It's being used to build small specialty models to solve problems rather than do it itself.
One example is HyInv, a protein inverse-folding model Hyra trained from scratch ...
🧠 1.96M parameters
⚡ 57× faster generation than ProteinMPNN
🚀 128 ~400-residue sequences in ~50 ms on one H20
But it didn't stop at models.
Hyra also optimized the open-source XiangShan RISC-V CPU
⚡ SPEC CPU2006 / GHz → +5.79%
📐 chip area → only +0.3%
It built a quantum-error-correction tool that Tencent says beat the previous qLDPC Challenge leader's encoding-efficiency metric by ~48%.
And designed a new robotic hand that eventually rotated 28.6% faster in simulation.
This is the part I find interesting:
🎯 Tencent says Hyra 2.0 is now participating in the pre-training, post-training, and inference-system optimization of its next-generation Hunyuan model.
Maybe giant agents increasingly build the tiny models, algorithms, and tools we run locally afterward.
That's a very different future for Local AI.
Link in ALT
We are entering a world of RSI.
Tencent's new harness trained a tiny 1.96M-parameter model from scratch that's 57× faster than the tool it was trying to replace.
And I think this is a ction for Local AI.
Tencent just introduced Hyra 2.0.
It's being used to build small specialty models to solve problems rather than do it itself.
One example is HyInv, a protein inverse-folding model Hyra trained from scratch ...
🧠 1.96M parameters
⚡ 57× faster generation than ProteinMPNN
🚀 128 ~400-residue sequences in ~50 ms on one H20
But it didn't stop at models.
Hyra also optimized the open-source XiangShan RISC-V CPU
⚡ SPEC CPU2006 / GHz → +5.79%
📐 chip area → only +0.3%
It built a quantum-error-correction tool that Tencent says beat the previous qLDPC Challenge leader's encoding-efficiency metric by ~48%.
And designed a new robotic hand that eventually rotated 28.6% faster in simulation.
This is the part I find interesting:
🎯 Tencent says Hyra 2.0 is now participating in the pre-training, post-training, and inference-system optimization of its next-generation Hunyuan model.
Maybe giant agents increasingly build the tiny models, algorithms, and tools we run locally afterward.
That's a very different future for Local AI.
Link in ALT
2
2
13
909
@aleckloss I'm with you. I have found great luck using Gemma 4 12B when coordinated and managed by smarter models like DeepSeek. My cost is NOT zero but pennies.
0
0
1
48
Here's a new model to quant. Meituan just gave us LongCat-2.5.
Stats
🧠 1.6 TRILLION parameters
⚡ ~48B active/token
📚 1M context
👁️ natively multimodal
🤖 built for long-horizon agents
⚙️ W4A4 INT4 inference
Check out the last bullet.
Raw weight math calcs
16-bit → ~3.2TB
~800GB
3-bit → ~600GB
2-bit → ~400GB
Yes. Even at 4-bit we're talking roughly 800GB of weights.
Meituan says its W4A4 implementation improves
📥 prefill → +43.3%
⚡ decode → +42.8%
⏱️ TTFT → -15.9%
versus W8A8.
And the jump from LongCat-2.0 is big.
Terminal Bench v4:
2.0 → 3.0
2.5 → 38.5
AutomationBench:
15.8 → 51.8
BrowseComp:
79.9 → 91.9
Same giant 1.6T / ~48B-active envelope. Much stronger agent.
The API is live, but I don't see the LongCat-2.5 weights on Hugging Face/GitHub yet.
Link in ALT
Stats
🧠 1.6 TRILLION parameters
⚡ ~48B active/token
📚 1M context
👁️ natively multimodal
🤖 built for long-horizon agents
⚙️ W4A4 INT4 inference
Check out the last bullet.
Raw weight math calcs
16-bit → ~3.2TB
~800GB
3-bit → ~600GB
2-bit → ~400GB
Yes. Even at 4-bit we're talking roughly 800GB of weights.
Meituan says its W4A4 implementation improves
📥 prefill → +43.3%
⚡ decode → +42.8%
⏱️ TTFT → -15.9%
versus W8A8.
And the jump from LongCat-2.0 is big.
Terminal Bench v4:
2.0 → 3.0
2.5 → 38.5
AutomationBench:
15.8 → 51.8
BrowseComp:
79.9 → 91.9
Same giant 1.6T / ~48B-active envelope. Much stronger agent.
The API is live, but I don't see the LongCat-2.5 weights on Hugging Face/GitHub yet.
Link in ALT
6
1
22
1,940
October '26 LLM list (final)
Fable 5.5 ❌
Fable 5.2 ❌
Haiku 5.5 ✅
GPT-6.1 Astra ✅
Meta Muse Spark 2 ❌
Meta Muse Spark 1.4 ❌
Meta Muse Spark Open ✅
Meta Must Wonder ✅
Meta Muse Wonder Full ✅
Meta Watermelon ❌
Gemini 3.6 Pro ❌
Gemini 4 Argon ✅
Gemini 4 Carbon ✅
GLM 5.5 Flash ❌
GLM 5.4 ❌
Kimi K4 ❌
Kimi K3.1✅
DeepSeek V4.1 Pro ❌
Qwen4 ❌
Qwen4-27B ❌
GLM-5.5 ❌
MiniMax M3 Pro ❌
MiniMax M3.5 ❌
Hunyuan3 MoE ❌
Mistral Large 4 ✅
Bonsai 2 2B, 4B, 9B ✅
Nemotron 3.5 Super VL ✅
Nemotron 3.5 Bolt 74B-A9B ✅
Step-5-Preview ✅
Fable 5.5 ❌
Fable 5.2 ❌
Haiku 5.5 ✅
GPT-6.1 Astra ✅
Meta Muse Spark 2 ❌
Meta Muse Spark 1.4 ❌
Meta Muse Spark Open ✅
Meta Must Wonder ✅
Meta Muse Wonder Full ✅
Meta Watermelon ❌
Gemini 3.6 Pro ❌
Gemini 4 Argon ✅
Gemini 4 Carbon ✅
GLM 5.5 Flash ❌
GLM 5.4 ❌
Kimi K4 ❌
Kimi K3.1✅
DeepSeek V4.1 Pro ❌
Qwen4 ❌
Qwen4-27B ❌
GLM-5.5 ❌
MiniMax M3 Pro ❌
MiniMax M3.5 ❌
Hunyuan3 MoE ❌
Mistral Large 4 ✅
Bonsai 2 2B, 4B, 9B ✅
Nemotron 3.5 Super VL ✅
Nemotron 3.5 Bolt 74B-A9B ✅
Step-5-Preview ✅
7
0
16
2,171
Your AI can run locally and still use online tools to make its responses even better.
For example, use an API to give it maps, add facts, photos, and even a budget slider.
That’s what one builder is doing with Frontier UI for Open WebUI.
The creator demonstrates a local day vacation itinerary with routes, photos, weather, and adjustable budgets—all inside the chat.
It supports
🗺️ Maps with walking and driving routes
📊 Interactive charts and sortable tables
🧮 Calculators and comparisons
📸 Photo galleries and place cards
📄 Word, Excel, PowerPoint and PDF exports through optional ArtCraft integrations
The model describes what to show. The tool fetches the information and builds the interface.
The creator uses Qwen3.8 Flash Next. Other models need suitable tool-calling support.
The core tool is one Python file added to Open WebUI. MIT licensed. Optional integrations require additional setup.
Local inference doesn’t mean everything is offline.
🔗 Links in ALT
For example, use an API to give it maps, add facts, photos, and even a budget slider.
That’s what one builder is doing with Frontier UI for Open WebUI.
The creator demonstrates a local day vacation itinerary with routes, photos, weather, and adjustable budgets—all inside the chat.
It supports
🗺️ Maps with walking and driving routes
📊 Interactive charts and sortable tables
🧮 Calculators and comparisons
📸 Photo galleries and place cards
📄 Word, Excel, PowerPoint and PDF exports through optional ArtCraft integrations
The model describes what to show. The tool fetches the information and builds the interface.
The creator uses Qwen3.8 Flash Next. Other models need suitable tool-calling support.
The core tool is one Python file added to Open WebUI. MIT licensed. Optional integrations require additional setup.
Local inference doesn’t mean everything is offline.
🔗 Links in ALT
5
1
6
1,007
Qwen hasn't announced Qwen4 just yet (next week), but Qwen's own coding client is already being prepped for ...
🧠 qwen4-plus
🧠 Qwen4-Max
Qwen Code merged PR #15987 on Oct. 31.
And the tests/configuration explicitly contain
qwen4-plus / Qwen4-Max
with reasoning controls
💭 high
Default → xhigh
The PR says the “qwen4 family” is joining Qwen's existing tiered DashScope reasoning-effort system and refers to the new model IDs as
“next-generation ids.”
This is Qwen's own repository preparing its client for Qwen4.
And now we know at least two names Qwen itself is preparing around
Qwen4-Plus
Qwen4-Max
with xHigh reasoning.
🔗 Link in ALT
🧠 qwen4-plus
🧠 Qwen4-Max
Qwen Code merged PR #15987 on Oct. 31.
And the tests/configuration explicitly contain
qwen4-plus / Qwen4-Max
with reasoning controls
💭 high
Default → xhigh
The PR says the “qwen4 family” is joining Qwen's existing tiered DashScope reasoning-effort system and refers to the new model IDs as
“next-generation ids.”
This is Qwen's own repository preparing its client for Qwen4.
And now we know at least two names Qwen itself is preparing around
Qwen4-Plus
Qwen4-Max
with xHigh reasoning.
🔗 Link in ALT
6
2
22
3,050
... and I thought modding a 2080 Ti was crazy. How about modding a RTX 5090 with 96GB of VRAM!
Greg Sky, a U.S. GPU rework specialist who already does high-memory NVIDIA conversions, posted
“64 and 96gb 5090 confirmed.”
And @LLMJunky says the big blocker has finally fallen
🔓 ked
Keep in mind that stock RTX 5090 has 🎮 32GB GDDR7
Potential mods could be
💾 64GB
🤯 96GB
A 96GB version would 3x the VRAM of a normal 5090 and match the memory capacity of an RTX PRO 6000 Blackwell (~$15K).
This part is crazy! 32 × 3GB GDDR7 modules and Greg says those 3GB modules are currently around $100 each.
So just the memory could be ~$3,200 before the custom board, RTX 5090, rework and labor.
⚠️ CAUTION!
We have NOT seen a working 96GB card yet.
No GPU-Z screenshot.
No benchmark.
No photos of a completed card.
No independent verification.
This is a very interesting claim from someone who actually works on these GPU mods but not a finished product yet.
Link in ALT
Greg Sky, a U.S. GPU rework specialist who already does high-memory NVIDIA conversions, posted
“64 and 96gb 5090 confirmed.”
And @LLMJunky says the big blocker has finally fallen
🔓 ked
Keep in mind that stock RTX 5090 has 🎮 32GB GDDR7
Potential mods could be
💾 64GB
🤯 96GB
A 96GB version would 3x the VRAM of a normal 5090 and match the memory capacity of an RTX PRO 6000 Blackwell (~$15K).
This part is crazy! 32 × 3GB GDDR7 modules and Greg says those 3GB modules are currently around $100 each.
So just the memory could be ~$3,200 before the custom board, RTX 5090, rework and labor.
⚠️ CAUTION!
We have NOT seen a working 96GB card yet.
No GPU-Z screenshot.
No benchmark.
No photos of a completed card.
No independent verification.
This is a very interesting claim from someone who actually works on these GPU mods but not a finished product yet.
Link in ALT
5
2
82
7,554
Custom inference engines are on the rise. Here's a new one.
Pithos is a C inference engine for Liquid AI’s LFM models.
The developer tested LFM2.5-24B-A2B Q4_K_M on
💻 Intel i5-10500H
🧠 16GB RAM
🎮 GTX 1650 Max-Q with 4GB VRAM
Reported generation speed
⚙️ llama.cpp CUDA: 17.5–21.9 tps
The model doesn’t fit entirely in that GPU’s memory.
Pithos keeps frequently used experts in VRAM, computes others on the CPU, and reads weights from disk when needed.
It also includes chat, an OpenAI-compatible server and agent tools.
These are developer-reported results from a specific llama.cpp build, with cooling controlled.
Consider that your next upgrade might be a better inference engine, rather than new HW.
🔗 Link in ALT.
Pithos is a C inference engine for Liquid AI’s LFM models.
The developer tested LFM2.5-24B-A2B Q4_K_M on
💻 Intel i5-10500H
🧠 16GB RAM
🎮 GTX 1650 Max-Q with 4GB VRAM
Reported generation speed
⚙️ llama.cpp CUDA: 17.5–21.9 tps
The model doesn’t fit entirely in that GPU’s memory.
Pithos keeps frequently used experts in VRAM, computes others on the CPU, and reads weights from disk when needed.
It also includes chat, an OpenAI-compatible server and agent tools.
These are developer-reported results from a specific llama.cpp build, with cooling controlled.
Consider that your next upgrade might be a better inference engine, rather than new HW.
🔗 Link in ALT.
14
29
243
9,784