882 tok/s prefill on M5 Max for Qwen3.8-27B, with uzu.
Compared to Splash:
prefill: 1.83x faster
time to first token: 1.81x faster
generation: 1.05x faster
Measured on MT-Bench, MATH-500 and HumanEval.
Compared to Splash:
prefill: 1.83x faster
time to first token: 1.81x faster
generation: 1.05x faster
Measured on MT-Bench, MATH-500 and HumanEval.
6
5
46
Today we are releasing our speculative decoding implementation in our inference engine uzu.
Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.
On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks.
Run the model:
Mirai-M: trymirai.com/local-models/a…
Mirai-L: trymirai.com/local-models/a…
Explore the benchmarks:
trymirai.com/metrics
Learn more about our speculative decoding implementation:
trymirai.com/blog/speculati…
Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators.
Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.
On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks.
Run the model:
Mirai-M: trymirai.com/local-models/a…
Mirai-L: trymirai.com/local-models/a…
Explore the benchmarks:
trymirai.com/metrics
Learn more about our speculative decoding implementation:
trymirai.com/blog/speculati…
Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators.
Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
30
61
463
📣 We've just published our latest research on state-of-the-art speculative decoding in interactivity.
Introducing a new approach that achieves a 4.37× speedup over autoregressive decoding in small batch setting and outperforms the strongest tuned DFlash baseline by 24.7%.
q6ewhLVs
github: github.com/trymirai/sglang
huggingface: huggingface.co/trymirai/weaver
Introducing a new approach that achieves a 4.37× speedup over autoregressive decoding in small batch setting and outperforms the strongest tuned DFlash baseline by 24.7%.
q6ewhLVs
github: github.com/trymirai/sglang
huggingface: huggingface.co/trymirai/weaver
0
4
21
If you've implemented speculative decoding, you've run into this: your speculator predicts the distribution correctly and still gets rejected.
Let’s take an example:
"The random number from 1 to 10 inclusive is"
Ideal LLM: uniform 1/10 across all numbers.
Good speculator: same situation, the naive scheme takes a random number from the speculator, then takes another random number from an LLM and can only accept the bonus token in 1/10 cases when they happen to match.
We should be able to always take its guess without the loss of generation quality.
The fix is straightforward.
Sample from both the speculator and the LLM using the same seed via the Gumbel-max trick.
Shared randomness means shared outcome when distributions match.
100% acceptance rate in this case. Near-optimal in practice.
We shipped this in github.com/trymirai/uzu
Full kernel implementation: trymirai.com/blog/how-to-im…
Let’s take an example:
"The random number from 1 to 10 inclusive is"
Ideal LLM: uniform 1/10 across all numbers.
Good speculator: same situation, the naive scheme takes a random number from the speculator, then takes another random number from an LLM and can only accept the bonus token in 1/10 cases when they happen to match.
We should be able to always take its guess without the loss of generation quality.
The fix is straightforward.
Sample from both the speculator and the LLM using the same seed via the Gumbel-max trick.
Shared randomness means shared outcome when distributions match.
100% acceptance rate in this case. Near-optimal in practice.
We shipped this in github.com/trymirai/uzu
Full kernel implementation: trymirai.com/blog/how-to-im…
0
1
21
LFM2.5-350M is now available on Mirai.
@liquidai smallest model outperforms Qwen3.5-0.8B on reasoning and agentic tool use.
Running on Mirai in full precision, it exceeds 70 tokens/second on iPhone.
@liquidai smallest model outperforms Qwen3.5-0.8B on reasoning and agentic tool use.
Running on Mirai in full precision, it exceeds 70 tokens/second on iPhone.
1
5
23
We tend to think AI quality = model quality.
Bigger model → better answers.
However, for local inference, performance depends on
model + hardware + execution together.
The same model can be much more or less efficient depending on how it’s run on-device.
That means local AI lem. It’s a systems problem. Of how efficiently can your model run on a real device.
That’s the shift.
Mirai is building the on-device inference layer,
the runtime between hardware and models, turning local compute into predictable, efficient, production-ready intelligence.
Bigger model → better answers.
However, for local inference, performance depends on
model + hardware + execution together.
The same model can be much more or less efficient depending on how it’s run on-device.
That means local AI lem. It’s a systems problem. Of how efficiently can your model run on a real device.
That’s the shift.
Mirai is building the on-device inference layer,
the runtime between hardware and models, turning local compute into predictable, efficient, production-ready intelligence.
1
0
11
AI isn’t moving from cloud → device.
It’s splitting.
Local models already handle a share of real-world queries, while frontier models remain essential for complex tasks.
It’s redistribution.
Simple workloads move closer to the user. Complex workloads stay in the cloud. The y default.
Once part of your AI stack runs locally:
• latency drops
• costs collapse
• privacy improves
• reliability increases
Mirai exists for this new architecture.
Where intelligence is split across layers,
but feels like one system.
It’s splitting.
Local models already handle a share of real-world queries, while frontier models remain essential for complex tasks.
It’s redistribution.
Simple workloads move closer to the user. Complex workloads stay in the cloud. The y default.
Once part of your AI stack runs locally:
• latency drops
• costs collapse
• privacy improves
• reliability increases
Mirai exists for this new architecture.
Where intelligence is split across layers,
but feels like one system.
0
1
12
The biggest constraint in AI isn’t models. It’s energy ⚡
Inference demand is exploding:
• Google Cloud saw a 1300× increase in token processing.
• NVIDIA reported 10× year-over-year growth
But data centers take years to build and require massive energy infrastructure.
So just:
“How do we build bigger models?”
It’s:
“How do we turn energy into intelligence more efficiently?”
That’s why intelligence-per-watt matters.
Just as performance-per-watt moved computing from mainframes to PCs, intelligence-per-watt may move AI from data centers to devices.
Mirai is building for that transition.
Inference demand is exploding:
• Google Cloud saw a 1300× increase in token processing.
• NVIDIA reported 10× year-over-year growth
But data centers take years to build and require massive energy infrastructure.
So just:
“How do we build bigger models?”
It’s:
“How do we turn energy into intelligence more efficiently?”
That’s why intelligence-per-watt matters.
Just as performance-per-watt moved computing from mainframes to PCs, intelligence-per-watt may move AI from data centers to devices.
Mirai is building for that transition.
1
0
9
AI today looks a lot like computing in the 1960s.
Massive centralized systems.
Shared infrastructure.
Users renting time on powerful machines.
Mainframes didn’t disappear because PCs became more powerful.
They disappeared because efficiency improved enough to run useful formance-per-watt doubled every ~1.5 years.
Now we’re seeing the same pattern in AI.
Local models are getting better.
Local hardware is getting faster.
Intelligence per watt is improving rapidly.
That’s how computing moved from mainframes to PCs.
And it may be how AI moves from data centers to devices.
Mirai is building for that transition.
Massive centralized systems.
Shared infrastructure.
Users renting time on powerful machines.
Mainframes didn’t disappear because PCs became more powerful.
They disappeared because efficiency improved enough to run useful formance-per-watt doubled every ~1.5 years.
Now we’re seeing the same pattern in AI.
Local models are getting better.
Local hardware is getting faster.
Intelligence per watt is improving rapidly.
That’s how computing moved from mainframes to PCs.
And it may be how AI moves from data centers to devices.
Mirai is building for that transition.
0
0
7
Computing progress used to be measured by raw performance. Then the metric shifted to performance per watt, and that shift moved us from mainframes to personal computers to smartphones.
AI is entering a similar phase.
The metric that matters isn't just model size or tokens per seful inference you get per unit of energy.
Why this matters:
– Data centers are energy-constrained,
– Most AI queries are lightweight enough to run locally,
– Devices already have powerful accelerators.
As on-device inference gets more efficient, it naturally moves closer to the user. The same pattern that moved computing from centralized systems to personal devices.
Mirai is building for that shift.
AI is entering a similar phase.
The metric that matters isn't just model size or tokens per seful inference you get per unit of energy.
Why this matters:
– Data centers are energy-constrained,
– Most AI queries are lightweight enough to run locally,
– Devices already have powerful accelerators.
As on-device inference gets more efficient, it naturally moves closer to the user. The same pattern that moved computing from centralized systems to personal devices.
Mirai is building for that shift.
1
0
11
Local inference is compounding from every direction:
• Models are getting better.
• Models are getting smaller.
• Hardware acceleration keeps improving.
• “Intelligence per watt” keeps rising.
When it crosses a critical threshold, inference becomes effectively zero cost, y owns the hardware.
We’re getting very close.
Mirai is building the runtime layer that turns this trend into infrastructure, so models can run predictably, efficiently, and production-ready on real devices.
Local isn’t the future.
It’s crossing the threshold now.
Learn more: trymirai.com
• Models are getting better.
• Models are getting smaller.
• Hardware acceleration keeps improving.
• “Intelligence per watt” keeps rising.
When it crosses a critical threshold, inference becomes effectively zero cost, y owns the hardware.
We’re getting very close.
Mirai is building the runtime layer that turns this trend into infrastructure, so models can run predictably, efficiently, and production-ready on real devices.
Local isn’t the future.
It’s crossing the threshold now.
Learn more: trymirai.com
9
0
14
Most AI doesn’t need hyperscale
Frontier labs should build frontier models.
But using the same infrastructure to answer: “write me an email” or “summarize this document” is architectural overkill.
Local LMs (≤20B parameters) now accurately answer ~88% of those queries, and proved 3.1× in two years.
So the real question becomes:
Why serve lightweight workloads with heavyweight infrastructure?
This isn’t about replacing frontier models.
It’s about matching the workload to the right execution layer.
That shift, from “biggest model everywhere”
to “right model, right place”,
is exactly what on-device inference enables.
That’s the layer we are building for.
Frontier labs should build frontier models.
But using the same infrastructure to answer: “write me an email” or “summarize this document” is architectural overkill.
Local LMs (≤20B parameters) now accurately answer ~88% of those queries, and proved 3.1× in two years.
So the real question becomes:
Why serve lightweight workloads with heavyweight infrastructure?
This isn’t about replacing frontier models.
It’s about matching the workload to the right execution layer.
That shift, from “biggest model everywhere”
to “right model, right place”,
is exactly what on-device inference enables.
That’s the layer we are building for.
6
0
18
We raised $10M led by @uncorkcap to make on-device AI inference accessible to developers building on Apple Silicon.
We're using the funds to expand model coverage across text, voice, and vision, working directly with model makers to bring their models on-device where latency, local execution over cloud.
Our proprietary inference engine is up to 37% faster than Apple's MLX with up to 59% faster prefill.
Developers integrate it in a few lines of code without needing specialized systems expertise.
The runtime supports hybrid deployments, routing inference between device and cloud seamlessly.
Models running on-device are becoming a new capability layer where developers can build system-level experiences independently.
We're building the infrastructure that enables that future.
The round includes participation from an incredible group of angels including @dps @FrancoisChauba1
@marcinzukowski @matiii @gokulr @scooterbraun @krishnanvijay @benparr @MattPRD @adityajami
and others.
More on it at our blog trymirai.com/blog/mirai-rai…
We're using the funds to expand model coverage across text, voice, and vision, working directly with model makers to bring their models on-device where latency, local execution over cloud.
Our proprietary inference engine is up to 37% faster than Apple's MLX with up to 59% faster prefill.
Developers integrate it in a few lines of code without needing specialized systems expertise.
The runtime supports hybrid deployments, routing inference between device and cloud seamlessly.
Models running on-device are becoming a new capability layer where developers can build system-level experiences independently.
We're building the infrastructure that enables that future.
The round includes participation from an incredible group of angels including @dps @FrancoisChauba1
@marcinzukowski @matiii @gokulr @scooterbraun @krishnanvijay @benparr @MattPRD @adityajami
and others.
More on it at our blog trymirai.com/blog/mirai-rai…
7
6
37
For years, “local models” meant compromise.
Not anymore.
-
Local models (≤20B):
• Solve 88.7% of single-turn chat & reasoning queries.
• Improved 3.1× in accuracy in just two years.
-
Local isn’t chasing frontier.
It’s covering the majority of everyday AI demand.
@trshift. To make smaller, capable models run predictably on real devices.
-
Detailed study about local models can be found here: hazyresearch.stanford.edu/blog/2025-11-1…
Not anymore.
-
Local models (≤20B):
• Solve 88.7% of single-turn chat & reasoning queries.
• Improved 3.1× in accuracy in just two years.
-
Local isn’t chasing frontier.
It’s covering the majority of everyday AI demand.
@trshift. To make smaller, capable models run predictably on real devices.
-
Detailed study about local models can be found here: hazyresearch.stanford.edu/blog/2025-11-1…
0
0
9
ChatGPT document, called "How People Use ChatGPT" shows that 77% of requests are:
1. practical guidance,
2. seeking Information,
3. writing.
Tasks, that do not require frontier model capabilities.
Which raises a simple question:
If most workloads are lightweight, why are we eaviest infrastructure?
Smaller models are already capable. The next shift isn’t bigger models. It’s matching the workload to the right layer.
That’s what on-device inference unlocks.
-
Link to the paper: cdn.openai.com/pdf/a253471f-8…
1. practical guidance,
2. seeking Information,
3. writing.
Tasks, that do not require frontier model capabilities.
Which raises a simple question:
If most workloads are lightweight, why are we eaviest infrastructure?
Smaller models are already capable. The next shift isn’t bigger models. It’s matching the workload to the right layer.
That’s what on-device inference unlocks.
-
Link to the paper: cdn.openai.com/pdf/a253471f-8…
0
0
11
The closer intelligence lives to the user, the more leverage it creates.
Local intelligence means:
• lower latency
• private user context
• predictable performance
• works even when the network doesn’t
That changes how products are built.
Instead of designing around round ign around state, continuity, and trust.
The cloud still matters.
But the most valuable intelligence is moving closer to the user.
We’re building the bridge between models and devices. So intelligence can live where it actually works.
Local intelligence means:
• lower latency
• private user context
• predictable performance
• works even when the network doesn’t
That changes how products are built.
Instead of designing around round ign around state, continuity, and trust.
The cloud still matters.
But the most valuable intelligence is moving closer to the user.
We’re building the bridge between models and devices. So intelligence can live where it actually works.
0
0
7
"Generic inference" isn't a thing.
The same LLM on CPU, GPU, or NPU is not the same model in practice.
Same weights, sure.
Same latency, power, throughput? No.
As @karpathy pointed out:
Model ≠ weights.
Model = weights + layout + kernels + quant + memory.
This is evice.
You're memory-bound and bandwidth-bound, not compute-bound.
We're moving from "better models" to "better model × runtime × hardware co-design" as the real moat.
At @trymirai we're working on exactly this.
The same LLM on CPU, GPU, or NPU is not the same model in practice.
Same weights, sure.
Same latency, power, throughput? No.
As @karpathy pointed out:
Model ≠ weights.
Model = weights + layout + kernels + quant + memory.
This is evice.
You're memory-bound and bandwidth-bound, not compute-bound.
We're moving from "better models" to "better model × runtime × hardware co-design" as the real moat.
At @trymirai we're working on exactly this.
0
0
6
The real AI stack isn’t just:
model → prompt → response.
In production, it looks like:
hardware → runtime → model → UX.
Hardware sets the limits.
The runtime decides latency, memory, and power.
The model operates inside those constraints.
Most teams start at the model and work backwards.
model → prompt → response.
In production, it looks like:
hardware → runtime → model → UX.
Hardware sets the limits.
The runtime decides latency, memory, and power.
The model operates inside those constraints.
Most teams start at the model and work backwards.
2
0
9
“It runs on device” is the new “it works on my machine.”
The hard part is making it:
• predictable
• debuggable
• energy-aware
• production-safe
That’s runtime engineering.
The hard part is making it:
• predictable
• debuggable
• energy-aware
• production-safe
That’s runtime engineering.
0
0
9
On-device AI is a 3-body problem:
model × runtime × hardware.
Predictable latency, power, and cost only exist if you control the runtime.
By runtime, we mean the inference engine:
- how weights are laid out,
- kernels executed,
- memory scheduled,
- and tokens streamed.
formance is won or lost.
model × runtime × hardware.
Predictable latency, power, and cost only exist if you control the runtime.
By runtime, we mean the inference engine:
- how weights are laid out,
- kernels executed,
- memory scheduled,
- and tokens streamed.
formance is won or lost.
1
0
11
Most AI startups think models are the product.
In reality, inference is the execution layer of AI software.
Whoever owns latency, memory, and routing owns the user experience.
That’s the layer we are building.
In reality, inference is the execution layer of AI software.
Whoever owns latency, memory, and routing owns the user experience.
That’s the layer we are building.
0
0
10
Users don’t care where your model runs.
They care how it feels.
Mirai makes real-time, device-native experiences that feel seamless for users.
⚡ Fast responses for text and audio.
🛜 Offline continuity. No network, no break.
📶 Consistent latency. Even under load.
They care how it feels.
Mirai makes real-time, device-native experiences that feel seamless for users.
⚡ Fast responses for text and audio.
🛜 Offline continuity. No network, no break.
📶 Consistent latency. Even under load.
0
0
8
Mirai is the only customizable on-device runtime that lets you run your own models directly on user devices.
With full control over tokens, logic, and pricing.
Cloud doesn’t go away.
Mirai just gives you a faster, local execution path.
With full control over tokens, logic, and pricing.
Cloud doesn’t go away.
Mirai just gives you a faster, local execution path.
0
0
8
Most inference engines are built for demos, not apps.
We built the Mirai inference engine from scratch for real-world use cases, not research demos.
- Hardware-aware execution (memory, scheduling, kernels).
- No cross-platform abstraction tax.
Learn more about us here: trymirai.com
We built the Mirai inference engine from scratch for real-world use cases, not research demos.
- Hardware-aware execution (memory, scheduling, kernels).
- No cross-platform abstraction tax.
Learn more about us here: trymirai.com
0
0
7
Mirai gives you full control over where and how your model runs, locally or in the cloud, using the same API.
No extra infra. Just smart routing logic you define.
Route by:
– Prompt length
– Latency constraints
– Device capability
– Confidence thresholds
– User segments or plan rompt-based hints, or assign model families per task type.
Mirai’s routing engine lets you tune for speed, privacy, or cost. On your terms.
No extra infra. Just smart routing logic you define.
Route by:
– Prompt length
– Latency constraints
– Device capability
– Confidence thresholds
– User segments or plan rompt-based hints, or assign model families per task type.
Mirai’s routing engine lets you tune for speed, privacy, or cost. On your terms.
0
0
8
The future of AI architecture is not “on-device 💻 vs cloud ☁️”. It’s hybrid by default.
- Train where scale is cheapest.
- Run where feedback is fastest.
- Route dynamically.
“Hybrid” only works if local execution is real, not a demo.
We are making that architecture viable in production.
- Train where scale is cheapest.
- Run where feedback is fastest.
- Route dynamically.
“Hybrid” only works if local execution is real, not a demo.
We are making that architecture viable in production.
0
0
10
Most AI workloads don’t need a data center.
They need:
- Low latency
- Predictable cost
- Offline resilience
- Local context
The missing piece isn’t another model. It’s execution.
Mirai exists to make local and hybrid inference a first-class option. Learn more: trymirai.com
They need:
- Low latency
- Predictable cost
- Offline resilience
- Local context
The missing piece isn’t another model. It’s execution.
Mirai exists to make local and hybrid inference a first-class option. Learn more: trymirai.com
1
0
8
Cloud-first ☁️ made sense when models were scarce. That era is ending.
Today: Train in the cloud. Run inference everywhere.
We’re building @trymirai to make this shift practical:
- train centrally,
- run inference directly on user devices,
- fall back to the cloud only when Learn more or contact us at: trymirai.com
Today: Train in the cloud. Run inference everywhere.
We’re building @trymirai to make this shift practical:
- train centrally,
- run inference directly on user devices,
- fall back to the cloud only when Learn more or contact us at: trymirai.com
0
0
7
Want your model to run directly on iPhones or Macs, without rewriting your backend?
Mirai lets model makers deploy full LLM pipelines on-device, using the same logic you already use in the cloud.
- No vendor lock.
- No infra rewrite.
👉 Learn more and connect with us at trymirai.com
Mirai lets model makers deploy full LLM pipelines on-device, using the same logic you already use in the cloud.
- No vendor lock.
- No infra rewrite.
👉 Learn more and connect with us at trymirai.com
0
0
9
We’re excited to announce our collaboration with @baseten
Now you can run both local and cloud models in one Mirai SDK, using the same API to route each request where it runs best.
Run local inference for speed and privacy, while freeing your cloud GPUs for what truly needs a wave isn’t cloud vs local – it’s both, working together.
@trymirai runs on-device. @baseten runs in the cloud. Hybrid is already here.
Try it now → trymirai.com
Now you can run both local and cloud models in one Mirai SDK, using the same API to route each request where it runs best.
Run local inference for speed and privacy, while freeing your cloud GPUs for what truly needs a wave isn’t cloud vs local – it’s both, working together.
@trymirai runs on-device. @baseten runs in the cloud. Hybrid is already here.
Try it now → trymirai.com
0
2
24