My lecture at MIT!✨

From Physics to Linear Algebra & Machine learning, I have learned a lot from MIT!

Yesterday, I had the honour of delivering a guest lecture on The state of AI Engineering, exploring:

- Prompt Engineering
- Retrieval Augmented Generation.
- Fine-Tuning Largd, how LLMs are bringing a paradigm shift

Thank you @RoyShilkrot for the invitation! 🙏

Grateful for the opportunity to engage with such brilliant minds!

If you are interested AI Engineering, I write a free weekly newsletter → @ML_Spring

12 KV caching techniques I’d learn to optimize LLM inference.

An LLM generates one token at a time, but each new token still needs information from earlier tokens. KV caching stores their keys and values so the model can reuse them instead of recomputing them at every step.

But an larger caches. More concurrent requests multiply that memory demand.

The graphic shows 12 techniques that change how this information is stored, shared, read, or moved.

→ 1. Sliding window

Keep recent tokens in layers that use local attention. Their caches stop growing once the window is full.

→ 2. Cross-layer sharing

Let multiple layers reuse shared keys and values, reducing duplicated storage. This requires a model architecture designed for sharing across layers.

→ 3. Multi-head latent attention (MLA)

Cache a compact representation instead of full per-head keys and values. Attention derives the information it needs from that representation.

→ 4. Hybrid Mamba

Combine attention with Mamba layers that maintain a fixed-size recurrent state. Those layers avoid a growing KV cache.

→ 5. Compressed attention

Combine token states into compressed representations, allowing attention to operate over fewer states. Compression can lose detail.

→ 6. Quantization

Store cached values using fewer bits. Each entry takes less memory, with possible accuracy and conversion costs.

→ 7. Prefix reuse

Reuse cached computation across requests with identical starting sequences. Shared system prompts or documents need less repeated processing.

→ 8. Sparse reads

Read selected cache blocks for each query. This reduces memory reads without necessarily shrinking the stored cache.

→ 9. Paging

Allocate growing caches in fixed-size blocks. This reduces wasted allocation space as requests grow and finish.

→ 10. Eviction

Remove selected token entries from the cache. This frees memory, but future attention loses direct access to those entries.

→ 11. Offloading

Move cache data outside GPU memory, such as into CPU memory. This frees GPU capacity, but transferring data back can add latency.

→ 12. Grouped-query and multi-query attention (GQA/MQA)

Share key/value heads across query heads. Fewer distinct keys and values need storing per token.

These techniques address different bottlenecks. Quantization reduces entry size, sparse reads reduce data access, and paging improves allocation.

Several can work together, but the right combination depends on the model and workload. Start with the bottleneck, then choose the techniques that address it.

I wrote the full breakdown of KV caching. The article is quoted below.

i just put Jev inside an open agent harness.

(150x cheaper decision-making than LLM)

agents burn a lot of tokens on decisions that are really just picking one option from a short list, like matching a client request to the right GitHub issue.

so i built a weekly client es exactly that. i tested it on 40 requests, and it was as accurate as letting Opus make every match, at 150x lower decision cost.

here's the tech stack i used:

→ TrueForge as the agent harness
→ Jev for the matching decisions
→ Daytona as the sandbox for code execution

Jev is a decision model from TypeSafe AI. it takes the options from your app and returns a structured choice with a calibrated confidence score, without generating any text.

that confidence decides what gets a second look. low-confidence matches go back to Opus for review, and in my test, that review caught 5 wrong matches that would otherwise have gone through.

TrueForge is TrueFoundry's harness, and it keeps context small. it loads tool definitions only when needed, writes large tool outputs to files, and lets the agent process data in scripts instead of the conversation.

on DevRev's Enterprise Bench, it solved about as many tasks as Claude Managed Agents on roughly a third of the tokens. it also keeps credentials outside the sandbox and holds every write for approval.

if you want to try this out, TrueForge is fully open source (MIT-licensed) and runs on your own machine with a single command.

TrueForge GitHub: github.com/truefoundry/tr…

(don't forget to star 🌟)

to help you reproduce my work and understand all the integration details, i wrote a detailed article. it's quoted below.

Speculative Decoding Clearly Explained

(done right, it can make LLM inference 4x faster)

A local LLM typically produces one token per model pass. Each pass reads the model’s weights from memory, often leaving compute units waiting for data.

Generating a sentence means e process for every token.

Now introduce a smaller model that can predict the next token quickly. Let it make several predictions ahead while the larger model waits to check them.

The smaller model is the draft model. The larger model, called the target model, is the LLM whose output we want to preserve.

The draft model proposes. The target model verifies.

Because the guesses are already available, the target can check several positions in one pass instead of generating each token separately.

Here’s how that works.

→ Draft a possible continuation.

After “The cat sat,” the draft model proposes “on the red mat.” These tokens are provisional.

→ Check the guesses together.

The target model scores all four positions in one forward pass. Each position uses the preceding draft tokens as context.

→ Accept from left to right.

For sampling, let p be the target model’s probability for a guess and q the draft model’s probability. Accept it with probability min(1, p/q), stopping at the first rejection.

→ Correct and continue.

Discard the rejected token and all later guesses. Sample a replacement from the positive difference between the target and draft distributions, normalized into probabilities.

In the diagram, “on” and “the” are accepted. “Red” is rejected, and the replacement is “big.”

That produces three new tokens from one target-model pass. If every guess is accepted, the target can supply an additional token.

This acceptance and correction procedure preserves the target model’s output distribution. The speedup comes from replacing expensive passes with cheaper drafting and shared verification.

Uzu, an open-source engine for Apple Silicon, takes this further. DFlash proposes candidates in parallel, Weaver builds coherent draft paths, and rollback-free verification checks branches without repeatedly restoring the model’s running state.

In our base M5 coding test, 4-bit Qwen3.5 9B reached 92 to 117 tokens per second with Uzu, compared with 22 using plain llama.cpp. These complete engine results also reflect quantization and hardware-specific kernels.

Explore Uzu on GitHub: github.com/trymirai/uzu

(don't forget to star 🌟)

My quoted article explains how Uzu combines these techniques and how to benchmark the result on your own prompts.

Speculative Decoding Clearly Explained

(done right, it can make LLM inference 4x faster)

A local LLM typically produces one token per model pass. Each pass reads the model’s weights from memory, often leaving compute units waiting for data.

Generating a sentence means e process for every token.

Now introduce a smaller model that can predict the next token quickly. Let it make several predictions ahead while the larger model waits to check them.

The smaller model is the draft model. The larger model, called the target model, is the LLM whose output we want to preserve.

The draft model proposes. The target model verifies.

Because the guesses are already available, the target can check several positions in one pass instead of generating each token separately.

Here’s how that works.

→ Draft a possible continuation.

After “The cat sat,” the draft model proposes “on the red mat.” These tokens are provisional.

→ Check the guesses together.

The target model scores all four positions in one forward pass. Each position uses the preceding draft tokens as context.

→ Accept from left to right.

For sampling, let p be the target model’s probability for a guess and q the draft model’s probability. Accept it with probability min(1, p/q), stopping at the first rejection.

→ Correct and continue.

Discard the rejected token and all later guesses. Sample a replacement from the positive difference between the target and draft distributions, normalized into probabilities.

In the diagram, “on” and “the” are accepted. “Red” is rejected, and the replacement is “big.”

That produces three new tokens from one target-model pass. If every guess is accepted, the target can supply an additional token.

This acceptance and correction procedure preserves the target model’s output distribution. The speedup comes from replacing expensive passes with cheaper drafting and shared verification.

Uzu, an open-source engine for Apple Silicon, takes this further. DFlash proposes candidates in parallel, Weaver builds coherent draft paths, and rollback-free verification checks branches without repeatedly restoring the model’s running state.

In our base M5 coding test, 4-bit Qwen3.5 9B reached 92 to 117 tokens per second with Uzu, compared with 22 using plain llama.cpp. These complete engine results also reflect quantization and hardware-specific kernels.

Explore Uzu on GitHub: github.com/trymirai/uzu

(don't forget to star 🌟)

My quoted article explains how Uzu combines these techniques and how to benchmark the result on your own prompts.

How to turn an LLM into a decision model?

Say we have a support ticket that says, “I was charged twice for invoice 4411. Please refund.”

Our application needs to make two decisions. Which team should handle the ticket, and whether to issue a refund.

An LM would generate text decision model would score the available choices directly.

Here’s how to turn an LLM into one:

→ Define the questions and choices.

Give the model the ticket alongside the decisions to make. For the team, the choices are billing, technical, and sales. For the refund, they are yes and no.

→ Keep the backbone and freeze its weights.

The backbone processes the input and builds internal representations of it. Add LoRA, small trainable components that adapt those representations while the original weights stay frozen.

→ Replace the language model head.

The language model head predicts the next token. Replace it with a decision head that gathers evidence from the ticket for each option, lets the questions and options interact, and produces one score per choice.

→ Convert scores into probabilities.

Apply softmax, which converts scores into probabilities, separately for each question. You get one probability distribution over the teams and another over the refund choices.

→ Train on the correct answers.

For this training example, the labels are billing and yes. Cross-entropy loss measures how well the predictions match those labels, and training updates only the decision head and LoRA components.

→ Calibrate on held-out examples.

After training, use examples excluded from training to adjust the model’s confidence. Its predicted probabilities should reflect how often those predictions are correct.

At inference time, one forward pass scores all the choices. Your application gets probabilities it can use to route the ticket and make the refund decision.

I trained a JEV-style decision model locally using Qwen3.5 0.8B. Accuracy went from 37% to 65% in just 60 steps, with 10 minutes of training on 4GB of VRAM.

Check out my YouTube video for the full walkthrough, from setup and training to evaluation and inference.

YouTube video: youtu.be/yKEesJaCwEc

(subscribe for more content like this)

I just trained my own Jev-style Decision model locally!

With Qwen3.5 0.8B as the base, accuracy jumped from 37% to 65% in just 60 steps (10 minutes of training).

All on just 4GB of VRAM.

This video shows how you can do the same.

Chapters:

0:00 - Intro
0:50 - Set up Unsloth del and dataset
5:19 - Train with LoRA
7:51 - Test it in the app
10:03 - Same workflow in code
12:38 - Train and evaluate (37% → 65%)
14:46 - Save and run inference
15:24 - Outro

If you’re new to JEV and want to understand how decision models work, I’ve written a detailed article about it. It’s quoted below.

Just published this on YouTube:
youtu.be/yKEesJaCwEc

This inference engine runs LLMs 4x faster than llama.cpp:

(Qwen3.5-9B at 92 tokens/s on 16GB Mac)

I ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time.

→ llama.cpp reached 22.0 tokens/s
→ MLX reached 25.1 tokens/s
→ Uzu reached 92.1 ws all three runs side by side.

The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding.

Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory.

A base M5 provides about 153 GB/s of memory bandwidth.

At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes.

Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon.

Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass.

Uzu pushes this further in two ways.

→ It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences.

→ It verifies several possible sequences as a tree, then commits state only for the path the full model accepts.

The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass.

In my run, every pass of the 9B model produced 7.5 output tokens on average.

Ordinary decoding produces exactly 1.

That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29.

The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path.

It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript.

The repository is here: github.com/trymirai/uzu

(don't forget to star 🌟)

I wrote the full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible results and code.

The article is quoted below.

A faster, 100% open-source alternative to Jev.

It’s called Laya.

Both models solve the same problem.

You give them unstructured state plus typed questions. Instead of generating a paragraph, they choose from answers you define and return probabilities your code can act on.
cisions differently.

Jev is a closed, hosted API. Its architecture and training are not public, but it works zero-shot and supports up to 255 options.

Laya is Apache 2.0 licensed, runs locally, and uses open-weight encoder models. Here is how its pipeline works.

→ It routes by language first. Plain Python selects the English or multilingual checkpoint before inference.

→ It creates one sequence for each question. Every possible answer gets a `[MASK]` marker. The question and state provide the context needed to compare those options.

→ It scores instead of generating. A bidirectional encoder and small decision head assign one score to each option. Softmax turns those scores into a probability distribution.

→ It batches the questions. They run together in one model call, although every question still carries its own copy of the state.

This removes network latency and token-by-token generation.

In one independent test, Laya took about 35 milliseconds on a local GPU. Jev took roughly 380 milliseconds over the network.

Laya also has no per-call API fee, keeps data on your infrastructure, and can run on older GPUs.

But it is not a drop-in replacement for every workload.

Jev is a hosted generalist that works well out of the box. Laya is a fast base you need to specialize.

Laya makes the most sense when you have a stable answer set and representative training data. Fine-tune it for the task, calibrate its probabilities on held-out examples, and send uncertain decisions to a stronger model or human reviewer.

Jev gives you stronger zero-shot decisions.

Laya gives you speed, privacy, control, and no per-call fee after you teach it your task.

The open-source repository is here: github.com/NandhaKishorM/…

If you want to self-host Laya, InstaCloud provides a really nice and easy way to do so.

Check this out: instacloud.com/templates/laya

If you're new to decision models, I wrote the full breakdown of how Jev. The article is quoted below.

LLM Routing, clearly explained!

(routing can cost more than not routing at all)

The basic idea sounds simple:

Send easy requests to cheaper models and reserve frontier models for difficult tasks.

But production routing is rarely that straightforward.

Every classification urpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior.

In this video, I cover:

→ Why DIY routing often breaks in production
→ How prefix caching changes the economics
→ Why agents need session pinning and model affinity
→ How Plano and Arch-Router handle intent resolution
→ How to build, test, and evaluate a router with DigitalOcean

Routing works, but only when the infrastructure accounts for the full session, not just the latest request.

The real question is not whether routing can reduce costs.

It is how much of your AI traffic has been overpaying this whole time.

Get started with Digital Ocean inference router: do.co/4i8JBrO

I recently wrote a detailed article on the same topic. The article is quoted below.

6 real-world use cases of Jev:

Many AI systems use generative models for decisions with a small, predefined answer space.

That is exactly where Jev fits. Give it the relevant state, define the questions, and receive typed decisions that your application can use directly.

Here to apply that pattern.

→ Input guardrails

Evaluate whether an incoming prompt is safe before passing it to an LLM. Jev can provide a focused semantic check while deterministic rules continue handling exact restrictions.

→ Model routing

Estimate how difficult a prompt is, then route it to the cheapest model capable of handling it. Straightforward requests can use a fast model while harder ones reach a frontier model.

→ Reranking

Judge which retrieved documents actually answer a query. The resulting relevance values can reorder candidates before the best context is sent downstream.

→ Tool-call gating

Decide whether a proposed action should run, require clarification, or be rejected. This semantic gate belongs after hard permission and policy checks, not in place of them.

→ Confidence gates

Evaluate whether a drafted response can be sent automatically, needs review, or should go directly to a person. The thresholds should be calibrated using representative examples before production use.

→ Output evaluations

Score a generated response against a defined rubric. Several focused criteria can share the same evidence, making it possible to evaluate grounding, relevance, honesty, or helpfulness without generating a written critique for every check.

The common thread is bounded judgment.

When your application needs a category, score, or probability from a known answer space, another round of open-ended text generation may be unnecessary.

Jev does not replace every LLM call. It gives us a focused primitive for the places where software needs a decision rather than another paragraph.

I explored the sixth use case in detail by building a Jev judge for agent evaluation and connecting it to an experiment workflow with Opik.

The full article is quoted below.

How vLLM stops one long prompt from stalling others:

(chunked prefill, clearly explained)

Every LLM request has two phases.

Prefill processes the complete input prompt in parallel. It is compute-heavy, builds the request’s KV cache, and produces the first output token.

Decode response one token at a time. Each step repeatedly reads model weights and the KV cache, making it primarily limited by memory bandwidth.

Now imagine Request A is already decoding and streaming tokens when Request B arrives with a 32K-token prompt.

Without chunked prefill, vLLM would process B’s complete prompt in one large pass. Request A cannot generate its next token during that work, so its response appears to freeze.

Chunked prefill splits B’s prompt into smaller token ranges. vLLM can process one chunk, give active requests another opportunity to decode, then continue B’s prefill across later iterations.

The same prompt still gets processed. It simply stops occupying the GPU as one uninterrupted block.

That introduces an important latency tradeoff.

→ Smaller chunks keep active responses moving.

They return control to the scheduler more frequently, reducing inter-token latency spikes for users already receiving an answer.

→ Larger chunks help the new request start sooner.

B cannot generate its first token until every prompt chunk has been processed. Larger chunks complete that work in fewer, more efficient passes, which usually improves time to first token.

→ Chunks that are too small add overhead.

Later chunks must attend to token states created by earlier chunks. Excessive splitting repeatedly reads more of the KV cache and can reduce GPU utilization.

→ Mixed batches can use the GPU more effectively.

Prefill is compute-heavy, while decode is memory-heavy. Scheduling prefill chunks alongside active decodes allows both kinds of work to share an iteration more efficiently.

There is no universal best chunk size.

Long prompts, many concurrent requests, and strict streaming targets usually favor smaller chunks. Workloads that prioritize time to first token may benefit from larger ones.

In vLLM V1, `--max-num-batched-tokens` controls how many tokens can be scheduled in an iteration. It is one of the main settings governing this tradeoff.

Chunked prefill does not reduce the work created by a long prompt. It prevents that work from deciding when everyone else receives their next token.

If you want to read more about this, I wrote the full breakdown on batching strategies in LLMs.

The article is quoted below.

HuggingFace just closed a major gap in harness engineering!

(the ultimate guide to multi-harness RL)

the same open-weight model can perform well inside one harness, then lose accuracy or produce invalid tool calls when moved to another.

this happens because training inside one model its specific tool names, output formats, context structure, and control flow. the model learns how to operate the harness, not just how to solve the task.

Hugging Face researchers tested a more portable approach called multi-harness reinforcement learning. instead of training a model through one agent interface, they trained it through Claude Code, Codex, OpenCode, and Mini-SWE-Agent.

to make that possible, they connected three open systems.

→ 𝗢𝗽𝗲𝗻𝗘𝗻𝘃 provides a standard interface between agent harnesses, reinforcement learning environments, and trainers. its capture proxy sits between the harness and model server, recording the exact tokens and generation probabilities needed for training.

→ 𝗛𝗮𝗿𝗯𝗼𝗿 runs agents against containerized tasks. it keeps the task, harness, and sandbox independent, allowing the same task to be attempted through different harnesses without rebuilding the environment.

→ 𝗧𝗥𝗟 is Hugging Face’s open-source library for post-training language models. it uses the trajectories captured by 𝗢𝗽𝗲𝗻𝗘𝗻𝘃 to update the model through reinforcement learning.

the real breakthrough is where the training data gets captured.

each harness keeps its native tools, prompts, context management, retries, and execution loop. the researchers do not recreate those behaviors inside the trainer. 𝗢𝗽𝗲𝗻𝗘𝗻𝘃 observes the model calls passing through each real harness and converts them into usable training sequences.

the team trained 𝗟𝗙𝗠𝟮.𝟱-𝟮.𝟲𝗕 across all four harnesses. the share of held-out tasks solved on the first attempt increased from 42.2% to 54.2%, with gains under every harness.

the trained model also used 31% fewer tool calls on tasks that both it and the base model solved.

training only in OpenCode improved the model too, but most of that gain stayed concentrated in OpenCode. multi-harness training spread the improvement across interfaces.

the experiment used one task family, one training seed, and unequal data exposure, so the results are not a universal ranking. even with those limitations, the mechanism matters.

open-source models cannot assume one deployment interface. if they need to work across harnesses, that portability must become part of training.

Read the full guide here: huggingface.co/spaces/FineEnv…

if you want to understand harness engineering and what an agent harness actually includes, i wrote a full breakdown to help you get started. the article is quoted below.

there is no best harness, only a best pairing (model, harness)

everyone is sleeping on this new JSON extraction model!

- open-weights
- matches Gemini Flash 3.5
- while being 3x faster

pulling structured data out of documents usually takes a chain of OCR, an LLM, and extra code to repair broken JSON.

lift, from @datalabto, does all of you give lift a PDF or image along with a JSON schema, and get back a JSON object that matches that schema.

the model reads every page of a document in one pass, so values that span multiple pages still come out right. any document you can describe in a schema works, from invoices to research papers.

here's what you get with lift:

- 90.2% field accuracy on 11,000 fields
- 9B model, nearly matches Gemini Flash 3.5 at 3x the speed
- built not to hallucinate; returns null if missing
- self-hostable with vLLM or Hugging Face

GitHub repo: github.com/datalab-to/lift

(don't forget to star 🌟)

OpenAI, Anthropic and Microsoft all back the same standard for running apps inside AI chats.

Booking .com, Figma, and Canva are already using it.

Still, most developers have never heard of it.

It's called MCP Apps.

So, why does it matter?

The next time someone buys from your visit it. They'll ask ChatGPT or Claude, and the model decides how your product shows up. Usually as plain text, sometimes wrong, with none of the experience you built.

That's the next shift in how people find you:

- SEO was about Google finding your page.
- AEO was about AI citing you.
- MCP Apps are about your actual product showing up inside the chat.

An MCP server returns data. An MCP App returns data plus an interactive UI, like hotel cards or an add-to-cart flow, right inside the chat. The model even knows what the user clicked.

In this episode, we build an MCP app from start to finish with Julien, the lead maintainer of Skybridge, an open-source framework that lets you turn any React app into an MCP app.

Chapters:

0:00 - Intro
2:44 - MCP servers vs MCP Apps
5:12 - Live demo: Booking .com in ChatGPT
9:24 - How MCP Apps work: context and shared state
15:43 - Skybridge and its devtools
17:48 - Live demo: an e-commerce app in ChatGPT
24:49 - Code walkthrough: MCP tools to React views
28:10 - Syncing UI state: "What's in my cart?"
32:16 - Agent skill to turn any React app → MCP App
37:52 - Outro

Link to Skybridge GitHub repo: github.com/alpic-ai/skybr…

MCP Apps bring your website into the chat. WebMCP does the reverse and lets AI agents use your website through tools it exposes. I've also written a full article on it, quoted below.