I wrote a CUDA inference engine from scratch that runs Qwen3.6-35B-A3B at 68 tokens/s on a 6 GB laptop GPU.
That's ~1.9x faster than hand-tuned llama.cpp on the same machine.
RTX 4050 laptop. No CUDA toolkit, no Python. Just the driver
That's ~1.9x faster than hand-tuned llama.cpp on the same machine.
RTX 4050 laptop. No CUDA toolkit, no Python. Just the driver
6
0
2
44
5/ I'm 18 and building in public. Next: more models, Linux, and squeezing even more out of consumer GPUs.
If you work on inference or AI chips and want this kind of thing on your hardware, DMs are open.
Repo: github.com/neerajdad123-b…
If you work on inference or AI chips and want this kind of thing on your hardware, DMs are open.
Repo: github.com/neerajdad123-b…
0
0
0
10
4/ It also ships an OpenAI-compatible server and imports GGUFs you already have (Q4_K experts carried over bit-for-bit, checked against llama.cpp with KL + top-1 agreement).
0
0
0
4
3/ On top of that: multi-token prediction. The model drafts the next tokens and verifies them in one pass. +20-35% on my laptop.
moex: 67.8 tok/s ik_llama.cpp: 36.0 llama.cpp: 33.3
moex: 67.8 tok/s ik_llama.cpp: 36.0 llama.cpp: 33.3
0
0
0
7
2/On top of that: multi-token prediction. The model drafts the next tokens and verifies them in one pass. +20-35% on my laptop.
moex: 67.8 tok/s ik_llama.cpp: 36.0 llama.cpp: 33.3
moex: 67.8 tok/s ik_llama.cpp: 36.0 llama.cpp: 33.3
0
0
0
10
The problem: a 35B MoE doesn't fit in 6 GB. But it only activates ~3B params per token.
So moex keeps the hot experts resident in VRAM and pages the rest from RAM, with an LRU policy I tuned on real routing traces from the model.
github.com/neerajdad123-b…
So moex keeps the hot experts resident in VRAM and pages the rest from RAM, with an LRU policy I tuned on real routing traces from the model.
github.com/neerajdad123-b…
0
0
1
60
neeraj retweeted
Every idea feels taken
Every API already exists
Every SaaS has 12 competitors
What are we even supposed to build anymore?
Every API already exists
Every SaaS has 12 competitors
What are we even supposed to build anymore?
519
32
1,548
147,470
neeraj retweeted
Growing on 𝕏 is really hard!! Apparently, some people have such big egos that they won't even follow you back when you follow them. 🤷
188
8
314
27,490
someones gonna stop him
OmarchyTV is out in beta
An #Omarchy style launcher for Android TV: themes, Waybar style bar, live stats and a terminal-style screensaver.
It comes with the omarchy-tv plugin, so your Omarchy PC can push themes and share its screen to the TV.
Based on tv-vibehack by @gathint. hh ❤️
It's on GitLab for now (my GitHub account is still stuck in review):
gitlab.com/piermark/omarc…
Wallpaper: Liftoff by @tcballard
0
0
0
12
