Today we are releasing our speculative decoding implementation in our inference engine uzu.

Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.

On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks.

Run the model:
Mirai-M: trymirai.com/local-models/a…
Mirai-L: trymirai.com/local-models/a…

Explore the benchmarks:
trymirai.com/metrics

Learn more about our speculative decoding implementation:
trymirai.com/blog/speculati…

Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators.

Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.

Today we are publishing our research essay: the computer should know the way.

Read it: trymirai.com/blog/the-compu…

About intent, latency, and why the layer that turns what you want into actions belongs on your own Mac. We at Mirai Labs are working on Mac automation. A person expresses an intention, and the system works out how to carry it out. Many of the intentions that matter most never appear in a request log, because the designed route is too long to bother.

We believe the computer should understand the intention, use all the context it has, adapt on the fly, and never interrupt the work.

We have released a new version of uzu and run the latest benchmarks on Qwen-3.8 27B we are by far the fastest at decode, and our checkpoints match Unsloth quality frontier and significantly surpass MLX ones in terms of quality

github.com/trymirai/uzu

Today we’re happy to show uzu.run, early preview of WebGPU support coming into uzu soon.

Try it: uzu.run

882 tok/s prefill on M5 Max for Qwen3.8-27B, with uzu.

Compared to Splash:

prefill: 1.83x faster
time to first token: 1.81x faster
generation: 1.05x faster

Measured on MT-Bench, MATH-500 and HumanEval.

For context on the prefill numbers, average prompt length per benchmark:

MT-Bench: 121 tokens
MATH-500: 124 tokens
HumanEval: 192 tokens

Please note that the evals used (MT-Bench, MATH-500 and HumanEval) are not very representative of the normal agentic use. We show the numbers on those specific datasets for the sake of interpretability of the results. The performance on the long context multi-turn interaction will be different.

We may have cooked a 2-bit version of Qwen3.8 27B which *actually works*, is not deep fried to benchmaxx, and fits in 8.5GB

We have not yet finished evaluating it, and it'll take some time to properly ship it, but if someone wants to try and share feedback please dm me

Mirai Labs (@trymirai) compressed Qwen3.8-27B to 2.4 bits. I ported it to llama.cpp and now it runs on a 12 GB card with 128K context.

With q4 KV cache the 128K window takes 11.3 GB of VRAM, with q8 KV you get 74K in 11.1 GB. Decode is 40 tok/s, still 34-35 tok/s at 62K deep, tok/s. I measured all of it on 1 RTX 3090 at 300 W, capped to what fits in 12 GB.

No requant anywhere. The GGUF carries Mirai's compressed codes bit for bit, and greedy output matches their own vLLM plugin.

Everything in the video was built by this same model as an agent in pi, 11 builds from 1 prompt each, a few needed a follow up and 1 is a blooper. That run was vLLM on my 3090s. Today the same weights run in llama.cpp.

Big thread, all links and recipes 👇

Our team loves to catch unique bugs

📣 We've just published our latest research on state-of-the-art speculative decoding in interactivity.

Introducing a new approach that achieves a 4.37× speedup over autoregressive decoding in small batch setting and outperforms the strongest tuned DFlash baseline by 24.7%.
q6ewhLVs
github: github.com/trymirai/sglang
huggingface: huggingface.co/trymirai/weaver

We are releasing our first quantized checkpoints for the Qwen3.5 series of models, co-designed jointly with our inference engine to achieve maximum possible performance on Apple hardware

Starting from 0.8B, 2B and 4B models

trymirai.com/blog/quantizat…

Google DeepMind released Gemma 4.
Our engineer @norpadon analyzed the architecture

If you've implemented speculative decoding, you've run into this: your speculator predicts the distribution correctly and still gets rejected.

Let’s take an example:
"The random number from 1 to 10 inclusive is"

Ideal LLM: uniform 1/10 across all numbers.
Good speculator: same situation, the naive scheme takes a random number from the speculator, then takes another random number from an LLM and can only accept the bonus token in 1/10 cases when they happen to match.

We should be able to always take its guess without the loss of generation quality.

The fix is straightforward.
Sample from both the speculator and the LLM using the same seed via the Gumbel-max trick.

Shared randomness means shared outcome when distributions match.
100% acceptance rate in this case. Near-optimal in practice.

We shipped this in github.com/trymirai/uzu
Full kernel implementation: trymirai.com/blog/how-to-im…

Day 0 on-device support of the latest and the smallest @liquidai model
LFM 2.5 350M is now available on @trymirai

LFM2.5-350M is now available on Mirai.

@liquidai smallest model outperforms Qwen3.5-0.8B on reasoning and agentic tool use.
Running on Mirai in full precision, it exceeds 70 tokens/second on iPhone.

@liquidai More on: trymirai.com/blog/lfm2.5-35…