Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about.

💡 I built this standalone .exe because businesses are already deploying local models to save money and ensure riendly - just drag drop and doubleclick)

My llama.cpp recipe 👉
🧠 Qwen3.5-4B Q4_K_M GGUF
⚙️ Ryzen 5 7540U — 6C/12T
🧵 --threads 9
🧵 --threads-batch 12
⚡ --prio 2
🔄 --poll 50
📦 --batch-size 2048
📦 --ubatch-size 512
🚀 --flash-attn on
🧠 KV cache: q4_0 / q4_0
🔧 --repack
💾 --mmap
👤 --parallel 1
🚫 --device none
🚫 --gpu-layers 0
🚫 KV/op GPU offload
🚫 MTP OFF

Interesting result 👉 MTP=3 was slower (~10 tps in benchmarks).

Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s.

That's about 10% faster just from tuning llama.cpp — on a basic laptop CPU.

30+B model release train continues.

A new 33B open model beat Qwen3.8-27B on 18 of 20 published benchmarks and its Q4 is only 19.1GB. 👈 👀

Nanbeige released Nanbeige4.3-33B.

Stats
🧠 33B dense
💾 Q4_K_S → 19.1GB
📚 512K context
🔓 Apache 2.0
🛠️ tool calling + thinking
cpp / Ollama / LM Studio / vLLM / SGLang

Agent benchmarks are notable

Nanbeige → Qwen3.8-27B

Terminal-Bench 2.1
80.9 → 73.0

Terminal-Bench 4.0
23.8 → 6.3

DeepSWE
65.5 → 42.5

Automation Bench:
54.2 → 38.3

JobBench
44.8 → 34.5

DeepPlanning
63.3 → 49.2

Nanbeige says it evaluated the models with the same inference hyperparameters and harnesses.

The 19.1GB Q4 weights can fit on a 24GB GPU but not at 512K context won't.

This is a conventional dense Llama architecture with a growing KV cache, and early community testing is already showing performance falling as context fills.

Holy 💩! Modders are unlocking hidden AI compute Nvidia left at reduced rates on older GeForce cards.

And now we have real numbers.

The open-source GPU Unlock projects change low-level speed selectors on RTX 20/30/40 cards before NVIDIA’s normal driver loads.

No VBIOS flash.
compute
87.5 → 157.4 TFLOPS

TF32
42.6 → 82.6 TFLOPS

And an RTX 4090 has now demonstrated roughly
165 → 330 TFLOPS

for FP16/BF16 Tensor operations with FP32 accumulation.

Unfortunately this does not make LLM generation 2× faster.

Real Local AI tests are showing the real results.

RTX 3090 / vLLM
📥 LLM prefill → +33%
⏱️ TTFT → ~24% lower
💬 decode → basically unchanged

A newer ~141K-context Qwen3.8-27B test

📥 prefill → +43%
💬 short-context decode → ~1%

And BF16 image-generation tests on a 3090 Ti reportedly gained roughly

🎨 MOSS Image → +59%
🎨 Qwen-Image 2.1 Turbo → +57%
🎨 Z-Image Turbo → +56%

Local LoRA training tests are reporting up to ~49% faster too.

This is because prompt processing / training / diffusion can be compute-bound.

LLM token generation is usually memory-bandwidth-bound, so the extra Tensor compute doesn’t help nearly as much.

And support is spreading
RTX 20 → experimental Turing unlock
RTX 30 → Ampere GPU Unlock
RTX 40 → Ada GPU Unlock

⚠️ Experimental. Primarily Linux today. Windows work is underway.

Link in ALT

Apple Silicon localmaxxers are attacking the Mac Dead-eGPU narrative.

Someone is booting Omarchy Linux directly on Apple Silicon Macs and running local LLMs on external AMD, Intel and NVIDIA GPUs.

On an M1 Max MacBook Pro, an external
🟦 Intel Arc Pro B60
💾 24GB VRAM

is irely on the eGPU at
⚡ 17.3 tps decode
📥 428 tps prefill

All 66 model layers are on the Arc card.

On an M2 Max MacBook Pro, an external RX 7600 8GB is also running llama.cpp/Vulkan.

Qwen2.5-7B⚡ ~29–32 tps decode

And after tuning, another 7B benchmark reached⚡ 56.3 tps

That's ~92.5% of the referenced native-PCIe result.

And it doesn't stop there.

An RTX 5060 Ti 16GB is working on the M1 Max through both CUDA and Vulkan

Qwen3.8-27B
🟢 CUDA → 24.6 tps
🟣 Vulkan → 22.5 tps

Check this out
The developer has now split Llama 3.3 70B across TWO different eGPU vendors on the same Apple Silicon MacBook

🟦 Intel Arc Pro B60+🟢 RTX 5060 Ti

⚠️ this is Linux on Apple Silicon, not macOS
⚠️ several required patches are not upstream yet
⚠️ GPUs currently need to be connected at boot

Link in thread

Qwen4-27B isn't released yet... but the inference stack may have leaked.👀

A newly merged XPU kernel PR explicitly names:

👉 Qwen4-27B dense

And it's already optimizing the unreleased model for these stats
⚡ FP8 → 2 GPUs
🧮 INT4 → 1 or 2 GPUs
🚀 MTP → 4 draft tokens
🧠 QSA are no public Qwen4-27B weights.
No official model card.

But the kernels expose enough shapes that the llama.cpp community has started reconstructing what the model may look like.

The current read looks like
🧠 27B dense
📐 hidden size → 5120
🔀 4 residual streams / hyper-connections
🎯 QSA sparse-attention kernels
🚀 MTP present
🧮 likely smaller FFN than Qwen3.8-27B

the full-attention Q/KV layout appears similar to Qwen3.8-27B, so the KV cache may not get smaller.

Link in ALT

Gigabyte built a desktop motherboard with 2TB of ECC RAM and support for up to 9 GPUs.

This new MB, the WRX90 AI TOP is built around AMD Threadripper Pro 9000/7000 WX CPUs.

This thing is basically a local AI server MB but from home AI labs

Stats
🧠 Threadripper Pro
💾 up to channel memory
🎮 7 full-length PCIe 5.0 slots
⚡ 6× PCIe 5.0 x16 + 1× x8
🌐 dual 10GbE
🖥️ IPMI
🔌 USB4
🔗 2× MCIO 8i

And Gigabyte has already showed it off with 9 GPUs.

Seven on the motherboard.

Plus two more connected externally.

They even built an active cooler specifically for the giant bank of DDR5 DIMMs, which Gigabyte claims can reduce memory temperatures by as much as 43°C.

There’s no announced price or firm retail date yet.

Think of the Local AI possibilities

Link in ALT

CUDA + GDDR7 VRAM gets crazy at the top!

AORUS Thunderbolt eGPU listings on eBay
💾 RTX 5060 Ti → 16GB → $1,521
💾 RTX 5090 → 32GB → $8,000

So 2× the VRAM for 5× the asking price

Obviously, the 5090 is also a much faster GPU, so this isn't an apples-to-apples comparison. to move from 16GB → 32GB?

NVIDIA's cheaper 64GB DGX Spark goes on sale today for $4,999.👀

Same GB10 Grace Blackwell.

Half the unified memory.

The new configuration keeps
🧠 GB10 Grace Blackwell
💾 64GB unified memory
🔌 ConnectX-7
🐧 DGX OS
⚙️ full NVIDIA AI software stack

NVIDIA says one 64GB Spark o 100B parameters locally.

And if 64GB isn't enough, buy 2× 64GB Sparks for → 128GB pooled memory 🙄

using → direct ConnectX-7 connection

NVIDIA says two clustered units delivered up to 1.7× the performance of one in its Qwen3.8-27B test.

It also uses the usual Local AI stack
🦙 llama.cpp
🟢 Ollama
🚀 vLLM
🖥️ LM Studio
🔥 PyTorch + CUDA

Availability starts today through:
Acer | ASUS | Dell | Gigabyte | HP | MSI

Starting price:
💰 $4,999

Link in ALT

Wondering where some of the RTX 5090s are going?

Fresh footage from China shows retail RTX 5090s getting stripped down and rebuilt into dense AI server cards.

Come on NOT cool!

Workers are taking cards from brands like
🟢 NVIDIA Founders Edition
🟠 Gigabyte
🔴 ASUS
⚫ PNY
inal gaming coolers and transplanting the expensive parts
🧠 GB202 GPU
💾 32GB GDDR7
onto custom PCBs with compact dual-slot blower coolers.

Because giant 3–4 slot gaming cards are terrible for dense AI servers.

A dual-slot blower card lets you pack far more 5090 compute into one chassis.

So the transformation takes a 🎮 retail gaming card
⬇️ stripped for GPU + VRAM 🧩 custom server PCB
🌬️ dual-slot blower 🤖 AI rack

Consumer RTX 5090s are valuable enough for AI that factories are literally rebuilding them into server-friendly cards.

Link to wccftech article in ALT

I thought the Gorgon Halos might be overpriced, and sales might have stalled.

Nope. Microsoft has sold out all its Surface 128GB RTX Sparks, and Minisforum has sold all of its 192GB AMD Ryzen AI Max+ PRO.

MiniMax released the M3.1 Flash weights today at ~252GB.

Localmaxxers immediately started squeezing them.

And within hours we're already at
📦 Official checkpoint + DSpark → 252GB
🍎 MLX mixed 3-bit → ~196GB
⚡ EXL3 3.06 bpw → ~171GB
🦙 Q2_K GGUF → ~160GB

This is a 428B B total / 23B active per token
📚 1M context
👁️ text + image + video input
⚡ sparse attention throughout
🚀 DSpark speculative drafter included

The official 252GB checkpoint is already heavily quantized.

In fact, one new GGUF conversion shows:
Q2_K → 160GB
Q3_K_M → 208GB
Q4_K_M → 262GB

vLLM says M3.1's native quantization took the checkpoint from about
M3 MXFP8 → 451.5GB
M3.1 → 250.1GB

And vLLM shipped day-zero support with
💾 NVFP4 KV cache
⚙️ NVFP4 MoE
⚡ DSpark speculative decoding
🟢 NVIDIA Blackwell/Rubin
🔴 AMD MI350

Now let's see who actually gets this thing running well on consumer hardware.

Link in ALT

Okay this is another very cool open release.

Tencent's setup for its interactive 🌎 AI world model uses 8 GPUs but it also built a low-memory mode that runs on one 24GB RTX 4090. 💾 👈

Hy-Nomas remembers the world as you interact with it.

That makes it different from a typical enerate a world, then keep changing it by
📝 change objects and scenes with text
🏃 describe new actions
🎨 change styles
🎥 control the camera
🔄 continue generating across multiple turns

Each new segment uses cached context from what was generated before it.

Hy-Nomas' standard inference configuration uses 8-way sequence parallelism and defaults to 8 GPUs.
But...
👉 Tencent built a separate low-memory configuration specifically for a single RTX 4090.

To make it fit
🧠 DiT weights → INT8
💾 KV cache → INT8
🎞️ PixelUMM decoder → FP8
⚙️ DiT computation → BF16
🗂️ older history → archived in CPU RAM
🎯 up to 18 historical latent frames selected for GPU attention
🔄 text encoder, DiT + decoder → moved between CPU/GPU as needed

And it keeps the same basic generation setup
🎥 720 × 1280
🎞️ 24 FPS output video
⚡ 8 denoising steps

24 FPS is the video's playback frame rate, not generation speed.

As you would expect, there is a tradeoff.

The standard configuration can keep the full KV history.

The single-4090 configuration limits the history actively used on the GPU so VRAM usage stays bounded, while older history is archived in system RAM.

That system-RAM archive can still grow as the generated world gets longer.

And this is currently a real NVIDIA/Linux setup:
🐧 Linux
🟢 NVIDIA GPU
⚙️ CUDA
📦 Docker recommended

Tencent warns that CPU offload needs enough system RAM and adds transfer overhead.

Link in ALT

This is the most exciting thing about Kimi and that’s they’re about to give us a 2.8T parameter model to figure out how to run locally!

Moonshot announced Kimi K3.1 Pro.

Stats
🧠 2.8T total / 104B active
📚 1M context
🖼️ multimodal
🎮 Unreal + Unity + Godot coding
🧊 Blender + igh / max
🔓 full weights coming by Nov 4

And Moonshot says substantially more post-training compute pushes the giant K3 base model close to the strongest proprietary models.

Terminal-Bench 4.0
Claude Opus 5.5 → 66.4
Kimi K3.1 Pro → 65.7
GPT-6 Astra → 58.2

Vendor-reported and not perfectly apples-to-apples, so independent testing comes next.

But here’s the Local AI problem we’re about to get.

Theoretical weight-only math for 2.8T
16-bit → 5.6TB
8-bit → 2.8TB
4-bit → 1.4TB
3-bit → ~1.05TB
2-bit → ~700GB

8× DGX Spark gives a cluster 1TB of aggregate unified memory so there’s a chance it will run at home.

The K3.1 Pro weights aren’t downloadable yet.

But by November 4 the race begins to see who can squeeze a 2.8T model onto hardware that has absolutely no business running it?

Nvidia squeezed Bolt onto an 8GB Jetson Orin Nano.

It's controlling a robot using the robot's hardware (no cloud).

Nvidia released Nemotron 3.5 Bolt E4B 4GB today. It's the smallest Bolt model, built for local AI applications like
🗣️ local voice assistants
🤖 robots
🎮 🏠 device assistants
🛠️ multi-turn tool use

In Nvidia's Reachy Mini demo:
👁️ Cosmos 3 Edge handles vision
🧠 Bolt E4B handles conversation + decisions
🛠️ Bolt calls the robot's animation tools
And both models run locally on a Jetson Orin Nano with 8GB of memory.

Bolt E4B Stats
🧠 8B total / 4B effective
📚 64K context
🛠️ multi-turn + async tool calling
💭 thinking on/off
🌎 35 languages
🔓 Open weights
⚖️ OpenMDW 1.1

And it's already ready for the Local AI stack:
🦙 llama.cpp
🟢 Ollama
🖥️ LM Studio
🚀 vLLM + SGLang
🤖 Hermes
🦞 OpenClaw

Nvidia calls this Bolt E4B 4GB and says it can occupy as little as ~4GB in target workloads.

The official Q4_K_M GGUF currently listed on Hugging Face is 6.18GB.

Link in ALT

A 12B model built to transform AI-written text can run locally with as little as ~6.2GB peak model memory.

It takes an AI-written draft and make it read more naturally while trying to preserve the facts contained in the text

Humanizer 12B is a Gemma 4 12B fine-tune that can run d the local quants get pretty small:
💾 Q8_0 → 12.7GB / ~13.7GB peak memory
💾 Q4_K_M → 7.6GB / ~10GB peak
💾 Q3 QAT → 5.6GB / ~8GB peak
💾 IQ2_XS QAT → 3.9GB / ~6.2GB peak

The Q4, Q3 and 2-bit versions were quantization-aware trained/distilled rather than simply conventionally quantized.

It runs with:
🦙 llama.cpp
🟢 Ollama
🖥️ LM Studio
🍎 MLX
🚀 vLLM
📦 offline Windows + Mac app

And the code + weights are Apache 2.0.

The author reports evaluated on 210 held-out English drafts, the BF16 model had 95% judged human by Originality at its strictest setting.

Only 11 / 210 were flagged as AI.

On the downloadable Q8 model, a separate LLM fact judge found 376 / 420 rewrites with no detected factual problem.

Link in ALT

Are you managing your digital employees. This Local AI agent orchestration platform does it for you.

OpenBMB released StaffDeck 1.0.0-preview, an open-source platform for building and running AI digital employees.

And you can self-host the entire orchestration layer.

Point compatible model endpoint and create agents with there capabilities out of the box
👨‍💻 IT
👥 HR
💰 Finance
⚖️ Legal
🏢 Administration

Each employee can have its own
📚 knowledge bases
🧠 memory
🛠️ skills + tools
📋 persistent SOPs
⏰ scheduled tasks
🔌 API + MCP connections
🧾 full execution traces

The new release adds Harness v3 + a CLI, so these aren't just agents sitting in a web chat.

From your terminal you can:
→ start an employee task
→ let it execute asynchronously
→ inspect its status/events
→ retrieve the result later
→ hand work to a human
→ resume the workflow

And disconnecting from the CLI doesn't automatically kill an already-running job.

StaffDeck itself runs locally on:
🪟 Windows | 🍎 macOS | 🐧 Linux

And it only requires an OpenAI-compatible Chat Completions endpoint for the model.

So your agent stack can use a locally hosted compatible model server rather.

Link in ALT

Okay so imagine you're chatting with a local LLM that has allocated most of your GPU's VRAM. Then you launch a game.

Instead of killing the model, LlamaForge gives the VRAM back.

That's the idea behind a new experimental feature in LlamaForge called Elastic VRAM.

When another bing GPU memory
🎮 Game needs more VRAM
⬇️ LlamaForge moves model layers back to CPU
🧠 MoE experts get moved first
💬 Your existing conversation survives
⬆️ When VRAM becomes available again, layers move back to the GPU
The model gets slower while it's shrunk to CPU.
But it doesn't have to disappear completely just because you launched something else.
Underneath, LlamaForge is running upstream llama.cpp and uses llama.cpp's slot save/restore to preserve the chat while it rearranges where the model lives.

There's also a simpler Game Mode
Launch game → unload model
Quit game → reload model

Elastic VRAM is the more interesting version because the model stays available while relinquishing GPU memory.

Keep in mind that it is ...
⚠️ Experimental
🟢 NVIDIA only today
⚙️ Off by default
🧪 Not widely tested yet
🔓 LlamaForge → MIT / local

Link in ALT

Are you getting ready for Kimi K3.1?

It may already be testing in Arena under the anonymous name Kibitz.

And there are now multiple clues pointing back to Moonshot.

🧬 Community tokenizer/fingerprint tests reportedly behave like Kimi

🗣️ In some Arena tests, Kibitz identifies
🔀 Kimi Code's configuration now contains a model-switch target named k3.1

And the anonymous naming pattern is eye catching
K2.5 → Kiwido
K2.6 → Kiwire
K3 → Kivine
K3.1? → Kibitz

K3 itself reportedly appeared anonymously in Arena roughly 30 hours before release.

None of this confirms Kibitz = Kimi K3.1.

Moonshot has not announced anything officially yet.

But K3.1 itself looks increasingly real.

Kimi Code now literally contains a migration rule pointing existing K3 sessions toward k3.1.

If Moonshot follows K3 with open weights again, K3.1 could become the next ridiculous multi-GPU / multi-node Local AI project.

Links in ALT

A new decision model tied for #1 on Artificial Analysis' Decision Index and its 8 points ahead of Jev 1.14.

SubQ Snap 1.1 is built to make the decisions inside an AI agent.

Artificial Analysis Stats
🥇 SubQ Snap 1.1 → 65
🥇 d1-ultra → 65
🥇 Elephant 9B → 65
Jev 1.14 → 57
al advantage, its context
📚 384K context

Subquadratic says its Sparse Attention architecture reduces attention compute from quadratic to linear scaling with context.

Snap reads the context and returns probabilities over a fixed set of allowed decisions in a single forward pass.

Subquadratic reports these benchmarks
🧰 When2Call → 96.2
📉 81% fewer confident errors on its unanswerable-query test
📚 384K context
💰 $0.04 / 1M input tokens
🆓 no output-token charge

This isn't a local model today as Snap 1.1 is currently offered through Subquadratic's API.

Layer this between the prompt and the models

Link in ALT