Last week I released the first article in my hands-on 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀 𝗙𝗿𝗼𝗺 𝗦𝗰𝗿𝗮𝘁𝗰𝗵 series. It covers the tool use element of an Agent.

I wanted to do this for quite some time now and finally the time has arrived!

If you are using any orchestration applications, you might be abstracted away from what using a tool really means. I believe that understanding applications from the base building blocks is really important so we are implementing all of the abstractions from scratch.

After finishing the project you will learn:
➡️ What AI Agents are.
➡️ How the Tool usage in AI Agents actually works.
➡️ How to build a decorator wrapper that extracts relevant details from a Python function to be passed to the LLM via system prompt.
➡️ How to think about constructing effective system prompts that can be used for Agents.
➡️ How to build an Agent class that is able to plan and execute actions using provided Tools.

You can find the article here: newsletter.swirlai.com/p/building-ai-…

You can also find a Jupyter notebook and python scripts that complement the article in a GitHub repo (links in the article).

Youtube videos coming soon!

If you’re looking for something to dig your hands into between holidays be sure to take a look!

Second part of the series is coming in a few weeks.

Let me know if there are any issues following the project. Feel free to DM me or leave comments here or under the article 👇

#AI #LLM #MachineLearning

This is how you measure your AI system as an AI Engineer 👇

For regular software you would track metrics like uptime, error rate, p95 latency. However, they say little about whether the system is fast where users feel it, affordable at scale or correct.

Here are the metrics we systems. It is useful to group them by the question they answer:

𝟭. 𝗜𝘀 𝗶𝘁 𝗳𝗮𝘀𝘁? (𝗟𝗮𝘁𝗲𝗻𝗰𝘆)

➡️ Time to first token (TTFT): how long the user is exposed to a blank screen, the number that defines perceived latency.
➡️ Inter-token latency (ITL): how smoothly tokens stream after the first one.
➡️ End-to-end latency at p50 / p95 / p99, dominated by output length, track it per use case rather than globally.

𝟮. 𝗖𝗮𝗻 𝗶𝘁 𝘀𝗰𝗮𝗹𝗲? (𝗧𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁 𝗮𝗻𝗱 𝗰𝗼𝘀𝘁)

➡️ Tokens per second per user vs total system throughput, the two trade off against each other on the same hardware.
➡️ Input and output tokens per request to measure your unit economics.
➡️ Cache hit rate - prompt caching is often the technique that reduces cost the most.
➡️ Cost per successful task, not cost per request, a cheap request that fails is a waste.

𝟯. 𝗜𝘀 𝗶𝘁 𝗰𝗼𝗿𝗿𝗲𝗰𝘁? (𝗤𝘂𝗮𝗹𝗶𝘁𝘆)

➡️ Task success rate on a labeled eval set, re-run on every prompt or model change.
➡️ Groundedness for RAG - is the answer supported by the retrieved context.
➡️ Retrieval precision@k and recall@k - generation cannot fix what retrieval never surfaced.
➡️ LLM-as-judge scores over time, calibrated against human labels.
➡️ User feedback signals: thumbs, edits to generated output, free form feedback.

𝟰. 𝗗𝗼𝗲𝘀 𝗶𝘁 𝗵𝗼𝗹𝗱 𝘂𝗽? (𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆)

➡️ Error, timeout and rate-limit rates per provider.
➡️ Retry and fallback rate - how often you silently switch to a backup model.
➡️ Guardrail trigger and refusal rates.

𝟱. 𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝘆𝗼𝘂𝗿 𝗮𝗴𝗲𝗻𝘁 𝗯𝗲𝗵𝗮𝘃𝗲? (𝗔𝗴𝗲𝗻𝘁 𝗺𝗲𝘁𝗿𝗶𝗰𝘀)

➡️ Tool-call error rate.
➡️ Steps and tokens per completed task - drift here means cost is rising while accuracy remains the same
➡️ Context window utilization - the early warning for compaction and truncation issues.

❗️ Latency and reliability show up on day one because standard infra emits them. Quality, cost per task, and agent behavior need deliberate instrumentation, and they are where AI systems fail in production.

Which metric caught a real problem for you that the standard dashboards missed? 👇

This is how you measure your AI system as an AI Engineer 👇

For regular software you would track metrics like uptime, error rate, p95 latency. However, they say little about whether the system is fast where users feel it, affordable at scale or correct.

Here are the metrics we systems. It is useful to group them by the question they answer:

𝟭. 𝗜𝘀 𝗶𝘁 𝗳𝗮𝘀𝘁? (𝗟𝗮𝘁𝗲𝗻𝗰𝘆)

➡️ Time to first token (TTFT): how long the user is exposed to a blank screen, the number that defines perceived latency.
➡️ Inter-token latency (ITL): how smoothly tokens stream after the first one.
➡️ End-to-end latency at p50 / p95 / p99, dominated by output length, track it per use case rather than globally.

𝟮. 𝗖𝗮𝗻 𝗶𝘁 𝘀𝗰𝗮𝗹𝗲? (𝗧𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁 𝗮𝗻𝗱 𝗰𝗼𝘀𝘁)

➡️ Tokens per second per user vs total system throughput, the two trade off against each other on the same hardware.
➡️ Input and output tokens per request to measure your unit economics.
➡️ Cache hit rate - prompt caching is often the technique that reduces cost the most.
➡️ Cost per successful task, not cost per request, a cheap request that fails is a waste.

𝟯. 𝗜𝘀 𝗶𝘁 𝗰𝗼𝗿𝗿𝗲𝗰𝘁? (𝗤𝘂𝗮𝗹𝗶𝘁𝘆)

➡️ Task success rate on a labeled eval set, re-run on every prompt or model change.
➡️ Groundedness for RAG - is the answer supported by the retrieved context.
➡️ Retrieval precision@k and recall@k - generation cannot fix what retrieval never surfaced.
➡️ LLM-as-judge scores over time, calibrated against human labels.
➡️ User feedback signals: thumbs, edits to generated output, free form feedback.

𝟰. 𝗗𝗼𝗲𝘀 𝗶𝘁 𝗵𝗼𝗹𝗱 𝘂𝗽? (𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆)

➡️ Error, timeout and rate-limit rates per provider.
➡️ Retry and fallback rate - how often you silently switch to a backup model.
➡️ Guardrail trigger and refusal rates.

𝟱. 𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝘆𝗼𝘂𝗿 𝗮𝗴𝗲𝗻𝘁 𝗯𝗲𝗵𝗮𝘃𝗲? (𝗔𝗴𝗲𝗻𝘁 𝗺𝗲𝘁𝗿𝗶𝗰𝘀)

➡️ Tool-call error rate.
➡️ Steps and tokens per completed task - drift here means cost is rising while accuracy remains the same
➡️ Context window utilization - the early warning for compaction and truncation issues.

❗️ Latency and reliability show up on day one because standard infra emits them. Quality, cost per task, and agent behavior need deliberate instrumentation, and they are where AI systems fail in production.

Which metric caught a real problem for you that the standard dashboards missed? 👇

𝗡𝗶𝗻𝗲 𝗽𝗹𝗮𝗰𝗲𝘀 𝘁𝗼 𝘂𝘀𝗲 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗝𝗲𝘃) 𝗶𝗻 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 𝘀𝘆𝘀𝘁𝗲𝗺 front to back 👇

𝟭. 𝗠𝗼𝗱𝗲𝗹 𝗿𝗼𝘂𝘁𝗶𝗻𝗴. The prompt goes to a decision model, it picks the tier. Cheap models for lookups, expensive for what you cannot afford 𝗮𝗶𝗹𝘀. A separate yes-or-no per potential hazard. Before the LLM sees the input or on the answer that has been produced.
𝟯. 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝘀𝘂𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆. Is what came back enough to answer, or should we search again.
𝟰. 𝗣𝗮𝘀𝘀𝗮𝗴𝗲 𝗳𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗮𝗻𝗱 𝗿𝗲𝗿𝗮𝗻𝗸𝗶𝗻𝗴. Does this passage answer the question or is it only keyword-adjacent. Apply per candidate and sort by probability.
𝟱. 𝗖𝗶𝘁𝗮𝘁𝗶𝗼𝗻 𝗰𝗵𝗲𝗰𝗸𝗶𝗻𝗴. Does this source support the claim, contradict it, or fail to address it.
𝟲. 𝗧𝗼𝗼𝗹-𝗰𝗮𝗹𝗹 𝗴𝗮𝘁𝗶𝗻𝗴. Allow, ask or deny per call, with the bands in your code.
𝟳. 𝗟𝗼𝗼𝗽 𝘁𝗲𝗿𝗺𝗶𝗻𝗮𝘁𝗶𝗼𝗻. Does the result actually satisfy the acceptance criteria?
𝟴. 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗰𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻. Score each piece of context and drop the stale ones instead of summarising them.
𝟵. 𝗘𝘃𝗮𝗹𝘀 𝗮𝘁 𝗳𝘂𝗹𝗹 𝗰𝗼𝘃𝗲𝗿𝗮𝗴𝗲. Significantly cheaper per verdict than an LLM judge. Potentially unlocks the ability to grade every trace instead of sampling.

Outside the Agentic RAG loop there are so many other use cases:

➡️ Bulk labeling.
➡️ Inbox triage.
➡️ Secret detection in diffs.
➡️ …

What to watch out for:

❌ You still need to eval your decision model use.
❌ Closely monitor your thresholds and version them as code.
❌ Decision models do not return an explanation, only which outputs are bad. If you require an explanation, fall back to an LLM judge.

👉 Generation is one call in most systems. The decisions around it are the other twenty, and that is where this pays.

What else belongs on the list? 👇

𝗡𝗶𝗻𝗲 𝗽𝗹𝗮𝗰𝗲𝘀 𝘁𝗼 𝘂𝘀𝗲 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗝𝗲𝘃) 𝗶𝗻 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 𝘀𝘆𝘀𝘁𝗲𝗺 front to back 👇

𝟭. 𝗠𝗼𝗱𝗲𝗹 𝗿𝗼𝘂𝘁𝗶𝗻𝗴. The prompt goes to a decision model, it picks the tier. Cheap models for lookups, expensive for what you cannot afford 𝗮𝗶𝗹𝘀. A separate yes-or-no per potential hazard. Before the LLM sees the input or on the answer that has been produced.
𝟯. 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝘀𝘂𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆. Is what came back enough to answer, or should we search again.
𝟰. 𝗣𝗮𝘀𝘀𝗮𝗴𝗲 𝗳𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗮𝗻𝗱 𝗿𝗲𝗿𝗮𝗻𝗸𝗶𝗻𝗴. Does this passage answer the question or is it only keyword-adjacent. Apply per candidate and sort by probability.
𝟱. 𝗖𝗶𝘁𝗮𝘁𝗶𝗼𝗻 𝗰𝗵𝗲𝗰𝗸𝗶𝗻𝗴. Does this source support the claim, contradict it, or fail to address it.
𝟲. 𝗧𝗼𝗼𝗹-𝗰𝗮𝗹𝗹 𝗴𝗮𝘁𝗶𝗻𝗴. Allow, ask or deny per call, with the bands in your code.
𝟳. 𝗟𝗼𝗼𝗽 𝘁𝗲𝗿𝗺𝗶𝗻𝗮𝘁𝗶𝗼𝗻. Does the result actually satisfy the acceptance criteria?
𝟴. 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗰𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻. Score each piece of context and drop the stale ones instead of summarising them.
𝟵. 𝗘𝘃𝗮𝗹𝘀 𝗮𝘁 𝗳𝘂𝗹𝗹 𝗰𝗼𝘃𝗲𝗿𝗮𝗴𝗲. Significantly cheaper per verdict than an LLM judge. Potentially unlocks the ability to grade every trace instead of sampling.

Outside the Agentic RAG loop there are so many other use cases:

➡️ Bulk labeling.
➡️ Inbox triage.
➡️ Secret detection in diffs.
➡️ …

What to watch out for:

❌ You still need to eval your decision model use.
❌ Closely monitor your thresholds and version them as code.
❌ Decision models do not return an explanation, only which outputs are bad. If you require an explanation, fall back to an LLM judge.

👉 Generation is one call in most systems. The decisions around it are the other twenty, and that is where this pays.

What else belongs on the list? 👇

𝗖𝘂𝘁𝘁𝗶𝗻𝗴 𝗮𝗻 𝗔𝗜 𝗯𝗶𝗹𝗹 𝘂𝘀𝘂𝗮𝗹𝗹𝘆 𝘀𝘁𝗮𝗿𝘁𝘀 𝘄𝗶𝘁𝗵 𝘁𝗼𝗸𝗲𝗻 𝗰𝗮𝗽𝘀 𝗮𝗻𝗱 𝗰𝗵𝗲𝗮𝗽𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀, 𝘄𝗵𝗶𝗰𝗵 𝗹𝗼𝘄𝗲𝗿𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗮𝗹𝗼𝗻𝗴 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝘀𝗽𝗲𝗻𝗱.

There is a different place to look at first. In most share of the cost comes from the work the system has already done and is paying for again:

➡️ The same question arrives in different words, so an exact-match cache never sees it and the model answers it from scratch every time
➡️ The agent drops context between turns and sessions, so it re-retrieves what it had and asks the user for what it was already told
➡️ All of that state, cache entries, session history, memory vectors, features, has to stay fast, and memory is the most expensive thing in the rack

None of those are model problems. They are caching, memory and storage decisions, which is the part of an AI system that gets the least design attention and ramps up the spend significantly.

I have been reading through how Redis approaches this. They offer three mechanisms that tackle the above problems in a single platform:

✅ Semantic caching: embed the incoming prompt, search prior prompts by meaning, and serve the stored response when it is close enough. Redis reports cache hits up to 15x faster and token spend down by up to 70%
✅ Agent memory: live session state with a TTL, plus facts extracted out of conversations into long-term memory as vectors, with the extraction running in the background instead of in the request (hot) path
✅ Flex: one database moving both keys and values between RAM and NVMe SSD storage tiers. You can adjust RAM ratio 10% and 50% per workload. Redis states this costs up to 80% less per gigabyte than running all-RAM, with no code change

The part I find interesting is that memory architecture is usually treated as a quality feature, something that makes the agent feel coherent. It is also a cost lever - an agent that forgets incurs the price twice for the same context.

👉 Here is their cost optimisation guide: fandf.co/3TVKDxt

Huge shoutout to @Redisinc for partnering on this post!

What share of your production traffic is a repeat of a question you have already answered? I rarely run into a team that is actually measuring this! 👇

𝗖𝘂𝘁𝘁𝗶𝗻𝗴 𝗮𝗻 𝗔𝗜 𝗯𝗶𝗹𝗹 𝘂𝘀𝘂𝗮𝗹𝗹𝘆 𝘀𝘁𝗮𝗿𝘁𝘀 𝘄𝗶𝘁𝗵 𝘁𝗼𝗸𝗲𝗻 𝗰𝗮𝗽𝘀 𝗮𝗻𝗱 𝗰𝗵𝗲𝗮𝗽𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀, 𝘄𝗵𝗶𝗰𝗵 𝗹𝗼𝘄𝗲𝗿𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗮𝗹𝗼𝗻𝗴 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝘀𝗽𝗲𝗻𝗱.

There is a different place to look at first. In most share of the cost comes from the work the system has already done and is paying for again:

➡️ The same question arrives in different words, so an exact-match cache never sees it and the model answers it from scratch every time
➡️ The agent drops context between turns and sessions, so it re-retrieves what it had and asks the user for what it was already told
➡️ All of that state, cache entries, session history, memory vectors, features, has to stay fast, and memory is the most expensive thing in the rack

None of those are model problems. They are caching, memory and storage decisions, which is the part of an AI system that gets the least design attention and ramps up the spend significantly.

I have been reading through how Redis approaches this. They offer three mechanisms that tackle the above problems in a single platform:

✅ Semantic caching: embed the incoming prompt, search prior prompts by meaning, and serve the stored response when it is close enough. Redis reports cache hits up to 15x faster and token spend down by up to 70%
✅ Agent memory: live session state with a TTL, plus facts extracted out of conversations into long-term memory as vectors, with the extraction running in the background instead of in the request (hot) path
✅ Flex: one database moving both keys and values between RAM and NVMe SSD storage tiers. You can adjust RAM ratio 10% and 50% per workload. Redis states this costs up to 80% less per gigabyte than running all-RAM, with no code change

The part I find interesting is that memory architecture is usually treated as a quality feature, something that makes the agent feel coherent. It is also a cost lever - an agent that forgets incurs the price twice for the same context.

👉 Here is their cost optimisation guide: fandf.co/3TVKDxt

Huge shoutout to @Redisinc for partnering on this post!

What share of your production traffic is a repeat of a question you have already answered? I rarely run into a team that is actually measuring this! 👇

𝗧𝗼𝗼𝗹 𝗖𝗮𝗹𝗹𝗶𝗻𝗴 𝘃𝗶𝗮 𝗠𝗖𝗣 vs. implementing it 𝗡𝗮𝘁𝗶𝘃𝗲 𝘁𝗼 𝘁𝗵𝗲 𝗔𝗴𝗲𝗻𝘁. What's the difference?

Recently there have been many MCP vs. Function Calling articles floating around. Let’s simplify.

❗️ 𝗖𝗹𝗮𝗿𝗶𝗳𝘆𝗶𝗻𝗴 𝗼𝗻𝗲 𝗱𝗲𝘁𝗮𝗶𝗹: Function Calling Systems are almost the same thing. You can implement tools via functions, the only difference is that functions are usually used to enforce stricter structure to the input and output schema.

𝘜𝘴𝘪𝘯𝘨 𝘔𝘊𝘗 𝘵𝘰 𝘦𝘹𝘱𝘰𝘴𝘦 𝘵𝘰𝘰𝘭𝘴 𝘵𝘰 𝘺𝘰𝘶𝘳 𝘈𝘨𝘦𝘯𝘵𝘴:

ℹ️ In this case your Agent becomes an MCP Host and implements one or more MCP Clients to communicate with MCP Servers.

𝟭. User Query is passed to the Agent (usually a Python application).
𝟮. The application implements MCP Client and via it retrieves all available tools from the MCP servers.
𝟯. The list of available Tools is passed together with the User Query to a LLM via a prompt. The LLM figures out which tools need to be invoked and with what parameters.
𝟰. The Agent application communicates with the MCP server (via MCP Client) again and sends the tool execution request. After execution completes the Agent receives the required data.
𝟱. User Query is sent to the LLM together with the data retrieved by the Tool calls.
𝟲. The answer is constructed and returned to the user via the Agent.

𝘜𝘴𝘪𝘯𝘨 𝘕𝘢𝘵𝘪𝘷𝘦 𝘍𝘶𝘯𝘤𝘵𝘪𝘰𝘯 𝘊𝘢𝘭𝘭𝘪𝘯𝘨:

𝟭. User Query is passed to the Agent (usually a Python application).
𝟮. All of the available Functions/Tools are defined as part of the Agent code (procedural memory).
𝟯. The list of available Tools is passed together with the User Query to a LLM via a prompt. The LLM figures out which functions need to be invoked and with what parameters.
𝟰. The Agent application directly executes the functions.
𝟱. User Query is sent to the LLM together with the data retrieved after function execution.
𝟲. The answer is constructed and returned to the user via the Agent.

𝗠𝘆 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀:

❗️ From the trenches, it looks like we have place for both methods of tool calling, only standardise behind MCP servers if necessary.
❗️ I see more practitioners abandoning MCP due to the lack of flexibility and additional failure points.

Are you using MCP already to expose Tools to your Agents? Let me know in the comments. 👇

𝗧𝗼𝗼𝗹 𝗖𝗮𝗹𝗹𝗶𝗻𝗴 𝘃𝗶𝗮 𝗠𝗖𝗣 vs. implementing it 𝗡𝗮𝘁𝗶𝘃𝗲 𝘁𝗼 𝘁𝗵𝗲 𝗔𝗴𝗲𝗻𝘁. What's the difference?

Recently there have been many MCP vs. Function Calling articles floating around. Let’s simplify.

❗️ 𝗖𝗹𝗮𝗿𝗶𝗳𝘆𝗶𝗻𝗴 𝗼𝗻𝗲 𝗱𝗲𝘁𝗮𝗶𝗹: Function Calling Systems are almost the same thing. You can implement tools via functions, the only difference is that functions are usually used to enforce stricter structure to the input and output schema.

𝘜𝘴𝘪𝘯𝘨 𝘔𝘊𝘗 𝘵𝘰 𝘦𝘹𝘱𝘰𝘴𝘦 𝘵𝘰𝘰𝘭𝘴 𝘵𝘰 𝘺𝘰𝘶𝘳 𝘈𝘨𝘦𝘯𝘵𝘴:

ℹ️ In this case your Agent becomes an MCP Host and implements one or more MCP Clients to communicate with MCP Servers.

𝟭. User Query is passed to the Agent (usually a Python application).
𝟮. The application implements MCP Client and via it retrieves all available tools from the MCP servers.
𝟯. The list of available Tools is passed together with the User Query to a LLM via a prompt. The LLM figures out which tools need to be invoked and with what parameters.
𝟰. The Agent application communicates with the MCP server (via MCP Client) again and sends the tool execution request. After execution completes the Agent receives the required data.
𝟱. User Query is sent to the LLM together with the data retrieved by the Tool calls.
𝟲. The answer is constructed and returned to the user via the Agent.

𝘜𝘴𝘪𝘯𝘨 𝘕𝘢𝘵𝘪𝘷𝘦 𝘍𝘶𝘯𝘤𝘵𝘪𝘰𝘯 𝘊𝘢𝘭𝘭𝘪𝘯𝘨:

𝟭. User Query is passed to the Agent (usually a Python application).
𝟮. All of the available Functions/Tools are defined as part of the Agent code (procedural memory).
𝟯. The list of available Tools is passed together with the User Query to a LLM via a prompt. The LLM figures out which functions need to be invoked and with what parameters.
𝟰. The Agent application directly executes the functions.
𝟱. User Query is sent to the LLM together with the data retrieved after function execution.
𝟲. The answer is constructed and returned to the user via the Agent.

𝗠𝘆 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀:

❗️ From the trenches, it looks like we have place for both methods of tool calling, only standardise behind MCP servers if necessary.
❗️ I see more practitioners abandoning MCP due to the lack of flexibility and additional failure points.

Are you using MCP already to expose Tools to your Agents? Let me know in the comments. 👇

𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗺𝗼𝗱𝗲𝗹𝘀 (𝗝𝗲𝘃 𝗯𝗲𝗶𝗻𝗴 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲𝗺) 𝘀𝗵𝗶𝗽𝗽𝗲𝗱 𝗮𝘀 𝗮 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗰𝗮𝘁𝗲𝗴𝗼𝗿𝘆 𝗹𝗮𝘀𝘁 𝗺𝗼𝗻𝘁𝗵. Here is why they are different to LLMs even though both can achieve the same result 👇

In decision models, you pass state, text or ose answers you declare in code. You get back a probability over those answers. That’s it, and it is also the selling point, nothing to parse and validate.

On the other hand, asking an LLM for a bounded answer means:

𝟭. Option list and rubric go in the prompt
𝟮. The model generates tokens, one forward pass each
𝟯. You parse the string, validate the schema, retry on invalidation
𝟰. The answer set is enforced after the text already exists

A decision model on the other hand makes the answer space an input, bounded before inference runs. With one scoring pass, you get a probability for every option you declare. Basically, a typed value with confidence attached.

Pros:

✅ You get a distribution, not a label. Per-option probability allows you to write the decision policy once: act above a threshold, cheap second check in the middle band, escalate the tail.
✅ Latency that significantly reduces the total latency of the loop. Jev runs P50 at 0.21 s and P95 at 0.34 s, and OpenAI advertises Decisions API to be close to 150 ms against roughly 1.6 s for the same model answering normally.
✅ No cost on the output side. Jev is $0.042 per million input tokens with free output. Generally, output is the expensive half of a judgment.

What to be aware of:

❌ Calibrated is a vendor claim. Check it against your own labelled set calibrate thresholds in house.
❌ Accuracy degrades as irrelevant state grows. As always, keep your context as lean as possible.
❌ Option names are more important for Decision Heads than the rubrics explaining them. Make sure to name them accordingly. I.e. refund_full_no_return instead of option_a.

👉 Count the model calls in your agent loop, e.g.:

➡️ Routing
➡️ Is the retrieved context sufficient
➡️ Is the draft grounded
➡️ Do we escalate

All of these have finite answer sets and run as full LLM calls with schema on top today. However, only one really needs to generate some text.

❗️ There is still a lot of use case for complex structure outputs, not every problem can be solved with simple classifiers.

Looking forward to seeing decision models proliferate in architectures our clients are building!

Which decisions in your system are a full LLM call that did not need to be? 👇

𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗺𝗼𝗱𝗲𝗹𝘀 (𝗝𝗲𝘃 𝗯𝗲𝗶𝗻𝗴 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲𝗺) 𝘀𝗵𝗶𝗽𝗽𝗲𝗱 𝗮𝘀 𝗮 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗰𝗮𝘁𝗲𝗴𝗼𝗿𝘆 𝗹𝗮𝘀𝘁 𝗺𝗼𝗻𝘁𝗵. Here is why they are different to LLMs even though both can achieve the same result 👇

In decision models, you pass state, text or ose answers you declare in code. You get back a probability over those answers. That’s it, and it is also the selling point, nothing to parse and validate.

On the other hand, asking an LLM for a bounded answer means:

𝟭. Option list and rubric go in the prompt
𝟮. The model generates tokens, one forward pass each
𝟯. You parse the string, validate the schema, retry on invalidation
𝟰. The answer set is enforced after the text already exists

A decision model on the other hand makes the answer space an input, bounded before inference runs. With one scoring pass, you get a probability for every option you declare. Basically, a typed value with confidence attached.

Pros:

✅ You get a distribution, not a label. Per-option probability allows you to write the decision policy once: act above a threshold, cheap second check in the middle band, escalate the tail.
✅ Latency that significantly reduces the total latency of the loop. Jev runs P50 at 0.21 s and P95 at 0.34 s, and OpenAI advertises Decisions API to be close to 150 ms against roughly 1.6 s for the same model answering normally.
✅ No cost on the output side. Jev is $0.042 per million input tokens with free output. Generally, output is the expensive half of a judgment.

What to be aware of:

❌ Calibrated is a vendor claim. Check it against your own labelled set calibrate thresholds in house.
❌ Accuracy degrades as irrelevant state grows. As always, keep your context as lean as possible.
❌ Option names are more important for Decision Heads than the rubrics explaining them. Make sure to name them accordingly. I.e. refund_full_no_return instead of option_a.

👉 Count the model calls in your agent loop, e.g.:

➡️ Routing
➡️ Is the retrieved context sufficient
➡️ Is the draft grounded
➡️ Do we escalate

All of these have finite answer sets and run as full LLM calls with schema on top today. However, only one really needs to generate some text.

❗️ There is still a lot of use case for complex structure outputs, not every problem can be solved with simple classifiers.

Looking forward to seeing decision models proliferate in architectures our clients are building!

Which decisions in your system are a full LLM call that did not need to be? 👇

Integrating 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 Systems via 𝗠𝗖𝗣 👇

If you are building RAG systems and packing many data sources for retrieval, most likely there is some agency present at least at the data source selection for retrieval stage.

This is how MCP enriches the evolution of s in such case (𝘱𝘰𝘪𝘯𝘵 2.):

𝟭. Analysis of the user query: we pass the original user query to a LLM based Agent for analysis. This is where:

➡️ The original query can be rewritten, sometimes multiple times to create either a single or multiple queries to be passed down the pipeline.
➡️ The agent decides if additional data sources are required to answer the query.

𝟮. If additional data is required, the Retrieval step is triggered. We could tap into variety of data types, few examples:

➡️ Real time user data.
➡️ Internal documents that a user might be interested in.
➡️ Data available on the web.
➡️ …

𝗧𝗵𝗶𝘀 𝗶𝘀 𝘄𝗵𝗲𝗿𝗲 𝗠𝗖𝗣 𝗰𝗼𝗺𝗲𝘀 𝗶𝗻:

✅ Each data domain can manage their own MCP Servers. Exposing specific rules of how the data should be used.
✅ Security and compliance can be ensured on the Servel level for each domain.
✅ New data domains can be easily added to the MCP server pool in a standardised way with no Agent rewrite needed enabling decoupled evolution of the system in terms of 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗮𝗹, 𝗘𝗽𝗶𝘀𝗼𝗱𝗶𝗰 𝗮𝗻𝗱 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗠𝗲𝗺𝗼𝗿𝘆.
✅ Platform builders can expose their data in a standardised way to external consumers. Enabling easy access to data on the web.
✅ AI Engineers can continue to focus on the topology of the Agent.

Learn all of this Hands-On in my End-to-End AI Engineering Bootcamp: maven.com/swirl-ai/end-t… . Next Cohort kicking off October 19th!

🎁 Use code YEAREND15 for 15% off.

𝟯. Retrieved data is consolidated and Reranked by a more powerful model compared to regular embedder. Data points are significantly narrowed down.
𝟰. If there is no need for additional data, we try to compose the answer (or multiple answers or a set of actions) straight via an LLM.
𝟱. The answer gets analyzed, summarized and evaluated for correctness and relevance:

➡️ If the Agent decides that the answer is good enough, it gets returned to the user.
➡️ If the Agent decides that the answer needs improvement, we try to rewrite the user query and repeat the generation loop.

Are you using MCP in your Agentic RAG systems? Let me know about your experience in the comment section 👇

Integrating 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 Systems via 𝗠𝗖𝗣 👇

If you are building RAG systems and packing many data sources for retrieval, most likely there is some agency present at least at the data source selection for retrieval stage.

This is how MCP enriches the evolution of s in such case (𝘱𝘰𝘪𝘯𝘵 2.):

𝟭. Analysis of the user query: we pass the original user query to a LLM based Agent for analysis. This is where:

➡️ The original query can be rewritten, sometimes multiple times to create either a single or multiple queries to be passed down the pipeline.
➡️ The agent decides if additional data sources are required to answer the query.

𝟮. If additional data is required, the Retrieval step is triggered. We could tap into variety of data types, few examples:

➡️ Real time user data.
➡️ Internal documents that a user might be interested in.
➡️ Data available on the web.
➡️ …

𝗧𝗵𝗶𝘀 𝗶𝘀 𝘄𝗵𝗲𝗿𝗲 𝗠𝗖𝗣 𝗰𝗼𝗺𝗲𝘀 𝗶𝗻:

✅ Each data domain can manage their own MCP Servers. Exposing specific rules of how the data should be used.
✅ Security and compliance can be ensured on the Servel level for each domain.
✅ New data domains can be easily added to the MCP server pool in a standardised way with no Agent rewrite needed enabling decoupled evolution of the system in terms of 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗮𝗹, 𝗘𝗽𝗶𝘀𝗼𝗱𝗶𝗰 𝗮𝗻𝗱 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗠𝗲𝗺𝗼𝗿𝘆.
✅ Platform builders can expose their data in a standardised way to external consumers. Enabling easy access to data on the web.
✅ AI Engineers can continue to focus on the topology of the Agent.

Learn all of this Hands-On in my End-to-End AI Engineering Bootcamp: maven.com/swirl-ai/end-t… . Next Cohort kicking off October 19th!

🎁 Use code YEAREND15 for 15% off.

𝟯. Retrieved data is consolidated and Reranked by a more powerful model compared to regular embedder. Data points are significantly narrowed down.
𝟰. If there is no need for additional data, we try to compose the answer (or multiple answers or a set of actions) straight via an LLM.
𝟱. The answer gets analyzed, summarized and evaluated for correctness and relevance:

➡️ If the Agent decides that the answer is good enough, it gets returned to the user.
➡️ If the Agent decides that the answer needs improvement, we try to rewrite the user query and repeat the generation loop.

Are you using MCP in your Agentic RAG systems? Let me know about your experience in the comment section 👇

𝗠𝗖𝗣 plus 𝗔𝟮𝗔, here is how they complement each other 👇

Protocol wars continue to rage, let's understand how Googles A2A (Agent2Agent) protocol is different from MCP and how they complement each other (read till the end).

𝘔𝘰𝘷𝘪𝘯𝘨 𝘱𝘪𝘦𝘤𝘦𝘴 𝘪𝘯 𝘔𝘊𝘗:

𝟭. MCP LMs at the core that want to access data through MCP.

❗️ When combined with A2A, an Agent becomes MCP Host.

𝟮. MCP Client - Clients that maintain 1:1 connections with servers.
𝟯. MCP Server - Lightweight programs that each expose specific capabilities through the standardised Model Context Protocol.
𝟰. Local Data Sources - Your computer’s files, databases, and services that MCP servers can securely access.
𝟱. Remote Data Sources - External systems available over the internet (e.g., through APIs) that MCP servers can connect to.

𝘌𝘯𝘵𝘦𝘳 𝘈2𝘈:

Where MCP falls short, A2A tries to help. In multi-Agent applications where state is not necessarily shared

𝟲. Agents (MCP Hosts) would implement and communicate via A2A protocol, that enables:

➡️ Secure Collaboration.
➡️ Task and State Management.
➡️ User Experience Negotiation.
➡️ Capability discovery - similar to MCP tools.

Learn all of this Hands-On in my End-to-End AI Engineering Bootcamp: maven.com/swirl-ai/end-t… . Next Cohort kicking off October 19th!

🎁 Use code YEAREND20 for 20% off - valid for the next two weeks.

𝗛𝗼𝗻𝗲𝘀𝘁 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀:

❗️ Open protocols for Agent communication are important but they are just a piece of the picture, there will be a need for standards that govern all of the existing protocols and other missing pieces. E.g.

❓ How do we standardise tracing and Observability in multi-agent IOA systems?
❓ How do we retain the identity of the running job of an AI Agent instance if the communication standard is not unified in different parts of the pipeline?
❓ ...

✅ More on this in my future posts, stay tuned!

Let me know your thoughts in the comments. 👇

𝗠𝗖𝗣 plus 𝗔𝟮𝗔, here is how they complement each other 👇

Protocol wars continue to rage, let's understand how Googles A2A (Agent2Agent) protocol is different from MCP and how they complement each other (read till the end).

𝘔𝘰𝘷𝘪𝘯𝘨 𝘱𝘪𝘦𝘤𝘦𝘴 𝘪𝘯 𝘔𝘊𝘗:

𝟭. MCP LMs at the core that want to access data through MCP.

❗️ When combined with A2A, an Agent becomes MCP Host.

𝟮. MCP Client - Clients that maintain 1:1 connections with servers.
𝟯. MCP Server - Lightweight programs that each expose specific capabilities through the standardised Model Context Protocol.
𝟰. Local Data Sources - Your computer’s files, databases, and services that MCP servers can securely access.
𝟱. Remote Data Sources - External systems available over the internet (e.g., through APIs) that MCP servers can connect to.

𝘌𝘯𝘵𝘦𝘳 𝘈2𝘈:

Where MCP falls short, A2A tries to help. In multi-Agent applications where state is not necessarily shared

𝟲. Agents (MCP Hosts) would implement and communicate via A2A protocol, that enables:

➡️ Secure Collaboration.
➡️ Task and State Management.
➡️ User Experience Negotiation.
➡️ Capability discovery - similar to MCP tools.

Learn all of this Hands-On in my End-to-End AI Engineering Bootcamp: maven.com/swirl-ai/end-t… . Next Cohort kicking off October 19th!

🎁 Use code YEAREND20 for 20% off - valid for the next two weeks.

𝗛𝗼𝗻𝗲𝘀𝘁 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀:

❗️ Open protocols for Agent communication are important but they are just a piece of the picture, there will be a need for standards that govern all of the existing protocols and other missing pieces. E.g.

❓ How do we standardise tracing and Observability in multi-agent IOA systems?
❓ How do we retain the identity of the running job of an AI Agent instance if the communication standard is not unified in different parts of the pipeline?
❓ ...

✅ More on this in my future posts, stay tuned!

Let me know your thoughts in the comments. 👇

𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 is a must have in your tool belt as an AI Engineer. 𝗧𝗿𝗮𝗰𝗶𝗻𝗴 sits at the core of it, why is it important?

Tracing and instrumentation of software have been around for decades now. With AI systems resembling regular software even more, we are here as well (with a few key differences).

Let’s look into the process of tracing from a perspective of a naive RAG system.

𝘍𝘦𝘸 𝘥𝘦𝘧𝘪𝘯𝘪𝘵𝘪𝘰𝘯𝘴:

𝘼) An Orchestrator in the GenAI system application is the central piece of software that orchestrates the end-to-end process. Think of apps using LangChain, LlamaIndex or Haystack.
𝘽) Trace is the end-to-end application flow from the entry point till the answer is produced, it is composed of smaller pieces called spans.
𝘾) Span is a smaller piece of the application flow that represents an atomic action like a function call or a database query. They can be sequential, or run in parallel.

ℹ️ As part of span we capture general metadata like start and end time, inputs and outputs of the span. On top of this metadata we track information specific to the GenAI system elements.

What might a trace look like for a naive RAG system?

𝟭. A query that has been submitted to the chat application.
𝟮. The query is embedded into a vector.

✅ Additional metadata like input token count is persisted with the span so that we can estimate the cost of the procedure.

𝟯. ANN lookup performed against the Vector DB to retrieve the most relevant context.

✅ Additional metadata about the query is persisted as part of the span together with the retrieved pieces of context and their relevance.

𝟰. A prompt is constructed from the system prompt and retrieved context.
𝟱. The prompt is passed to the LLM to construct the answer.

✅ Additional metadata about input and output token count is captured together with the span so that we can estimate the cost of the procedure.

𝘞𝘩𝘺 𝘪𝘴 𝘵𝘳𝘢𝘤𝘪𝘯𝘨 𝘰𝘧 𝘎𝘦𝘯𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴 𝘪𝘮𝘱𝘰𝘳𝘵𝘢𝘯𝘵?

- These applications are usually complex chains, errors can happen in different steps of your application. E.g. Embedding of query is taking longer than expected or you have reached API limits of LLM provider.
- Cost for calling LLM APIs will be variable depending on the length of inputs and produced outputs. You would usually trace this information and analyze it to help forecast expenses.
- GenAI systems are non-deterministic and will deteriorate over time. They need to be evaluated on span level rather than input/output of the entire system so that you can tune each piece separately.
- …

Are you tracing your Agents? Let me know in the comments 👇

𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 is a must have in your tool belt as an AI Engineer. 𝗧𝗿𝗮𝗰𝗶𝗻𝗴 sits at the core of it, why is it important?

Tracing and instrumentation of software have been around for decades now. With AI systems resembling regular software even more, we are here as well (with a few key differences).

Let’s look into the process of tracing from a perspective of a naive RAG system.

𝘍𝘦𝘸 𝘥𝘦𝘧𝘪𝘯𝘪𝘵𝘪𝘰𝘯𝘴:

𝘼) An Orchestrator in the GenAI system application is the central piece of software that orchestrates the end-to-end process. Think of apps using LangChain, LlamaIndex or Haystack.
𝘽) Trace is the end-to-end application flow from the entry point till the answer is produced, it is composed of smaller pieces called spans.
𝘾) Span is a smaller piece of the application flow that represents an atomic action like a function call or a database query. They can be sequential, or run in parallel.

ℹ️ As part of span we capture general metadata like start and end time, inputs and outputs of the span. On top of this metadata we track information specific to the GenAI system elements.

What might a trace look like for a naive RAG system?

𝟭. A query that has been submitted to the chat application.
𝟮. The query is embedded into a vector.

✅ Additional metadata like input token count is persisted with the span so that we can estimate the cost of the procedure.

𝟯. ANN lookup performed against the Vector DB to retrieve the most relevant context.

✅ Additional metadata about the query is persisted as part of the span together with the retrieved pieces of context and their relevance.

𝟰. A prompt is constructed from the system prompt and retrieved context.
𝟱. The prompt is passed to the LLM to construct the answer.

✅ Additional metadata about input and output token count is captured together with the span so that we can estimate the cost of the procedure.

𝘞𝘩𝘺 𝘪𝘴 𝘵𝘳𝘢𝘤𝘪𝘯𝘨 𝘰𝘧 𝘎𝘦𝘯𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴 𝘪𝘮𝘱𝘰𝘳𝘵𝘢𝘯𝘵?

- These applications are usually complex chains, errors can happen in different steps of your application. E.g. Embedding of query is taking longer than expected or you have reached API limits of LLM provider.
- Cost for calling LLM APIs will be variable depending on the length of inputs and produced outputs. You would usually trace this information and analyze it to help forecast expenses.
- GenAI systems are non-deterministic and will deteriorate over time. They need to be evaluated on span level rather than input/output of the entire system so that you can tune each piece separately.
- …

Are you tracing your Agents? Let me know in the comments 👇

𝗠𝗖𝗣 plus 𝗔𝟮𝗔, here is how they complement each other 👇

Protocol wars continue to rage, let's understand how Googles A2A (Agent2Agent) protocol is different from MCP and how they complement each other (read till the end).

𝘔𝘰𝘷𝘪𝘯𝘨 𝘱𝘪𝘦𝘤𝘦𝘴 𝘪𝘯 𝘔𝘊𝘗:

𝟭. MCP LMs at the core that want to access data through MCP.

❗️ When combined with A2A, an Agent becomes MCP Host.

𝟮. MCP Client - Clients that maintain 1:1 connections with servers.
𝟯. MCP Server - Lightweight programs that each expose specific capabilities through the standardised Model Context Protocol.
𝟰. Local Data Sources - Your computer’s files, databases, and services that MCP servers can securely access.
𝟱. Remote Data Sources - External systems available over the internet (e.g., through APIs) that MCP servers can connect to.

𝘌𝘯𝘵𝘦𝘳 𝘈2𝘈:

Where MCP falls short, A2A tries to help. In multi-Agent applications where state is not necessarily shared

𝟲. Agents (MCP Hosts) would implement and communicate via A2A protocol, that enables:

➡️ Secure Collaboration.
➡️ Task and State Management.
➡️ User Experience Negotiation.
➡️ Capability discovery - similar to MCP tools.

𝗛𝗼𝗻𝗲𝘀𝘁 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀:

❗️ Open protocols for Agent communication are important but they are just a piece of the picture, there will be a need for standards that govern all of the existing protocols and other missing pieces. E.g.

❓ How do we standardise tracing and Observability in multi-agent IOA systems?
❓ How do we retain the identity of the running job of an AI Agent instance if the communication standard is not unified in different parts of the pipeline?
❓ ...

✅ More on this in my future posts, stay tuned!

Let me know your thoughts in the comments. 👇

𝗔𝗜 𝗳𝗲𝗮𝘁𝘂𝗿𝗲𝘀 𝗶𝗻 𝗮 𝗦𝗮𝗮𝗦 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗮𝗿𝗲 𝗮𝗻 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 𝗮𝘀 𝗺𝘂𝗰𝗵 𝗮𝘀 𝗮 𝗺𝗼𝗱𝗲𝗹 𝗽𝗿𝗼𝗯𝗹𝗲𝗺.

Every software company is wiring AI into its product right now and the work concentrates in the gap between a prototype and a product that serves thousands. This is what I mean:

➡️ Inference traffic is bursty and grows with every tenant you onboard
➡️ Tenant data has to stay isolated through every retrieval step and model call
➡️ Regulated buyers ask about SOC 2, HIPAA and ISO 27001 before they ask about features
➡️ Once the product works, you still need a path into enterprise procurement

This is where the 𝗔𝗪𝗦 𝗜𝗦𝗩 resources made sense for me, one platform covering the full scope from building to selling:

✅ Compute that matches your workload shape: Lambda for spiky event-driven paths, ECS or EKS for long-running multi-tenant services
👆 The mistake is picking one shape for the whole product. Inference paths and ingestion jobs have almost nothing in common except that they both run your code.

✅ Amazon Bedrock with over 100 foundation models, so AI features ship without a model-serving layer of your own

✅ A data layer designed for per-tenant isolation, from Aurora and DynamoDB to OpenSearch for retrieval
✅ Security primitives (IAM, VPC isolation, encryption at rest and in transit) that map directly onto the compliance controls regulated industries require
👆 These are the two teams defer but pay for later. Adding isolation after launch means reopening every query path and every retrieval call you've already shipped.

✅ AWS Marketplace and ISV Accelerate co-sell, which turn enterprise procurement into a channel instead of a grind

👉 See how software companies are building on AWS: fandf.co/3RSKMRf

Follow @awscloud for more on what ISVs are shipping on the platform.

𝑇ℎ𝑖𝑠 𝑝𝑜𝑠𝑡 𝑖𝑠 𝑠𝑝𝑜𝑛𝑠𝑜𝑟𝑒𝑑 𝑏𝑦 𝐴𝑊𝑆.

If you are building AI into a SaaS product, which part has been harder in practice: the model side, or the multi-tenant infrastructure around it? 👇

𝗔𝗜 𝗳𝗲𝗮𝘁𝘂𝗿𝗲𝘀 𝗶𝗻 𝗮 𝗦𝗮𝗮𝗦 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗮𝗿𝗲 𝗮𝗻 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 𝗮𝘀 𝗺𝘂𝗰𝗵 𝗮𝘀 𝗮 𝗺𝗼𝗱𝗲𝗹 𝗽𝗿𝗼𝗯𝗹𝗲𝗺.

Every software company is wiring AI into its product right now and the work concentrates in the gap between a prototype and a product that serves thousands. This is what I mean:

➡️ Inference traffic is bursty and grows with every tenant you onboard
➡️ Tenant data has to stay isolated through every retrieval step and model call
➡️ Regulated buyers ask about SOC 2, HIPAA and ISO 27001 before they ask about features
➡️ Once the product works, you still need a path into enterprise procurement

This is where the 𝗔𝗪𝗦 𝗜𝗦𝗩 resources made sense for me, one platform covering the full scope from building to selling:

✅ Compute that matches your workload shape: Lambda for spiky event-driven paths, ECS or EKS for long-running multi-tenant services
👆 The mistake is picking one shape for the whole product. Inference paths and ingestion jobs have almost nothing in common except that they both run your code.

✅ Amazon Bedrock with over 100 foundation models, so AI features ship without a model-serving layer of your own

✅ A data layer designed for per-tenant isolation, from Aurora and DynamoDB to OpenSearch for retrieval
✅ Security primitives (IAM, VPC isolation, encryption at rest and in transit) that map directly onto the compliance controls regulated industries require
👆 These are the two teams defer but pay for later. Adding isolation after launch means reopening every query path and every retrieval call you've already shipped.

✅ AWS Marketplace and ISV Accelerate co-sell, which turn enterprise procurement into a channel instead of a grind

👉 See how software companies are building on AWS: fandf.co/3RSKMRf

Follow @awscloud for more on what ISVs are shipping on the platform.

𝑇ℎ𝑖𝑠 𝑝𝑜𝑠𝑡 𝑖𝑠 𝑠𝑝𝑜𝑛𝑠𝑜𝑟𝑒𝑑 𝑏𝑦 𝐴𝑊𝑆.

If you are building AI into a SaaS product, which part has been harder in practice: the model side, or the multi-tenant infrastructure around it? 👇

Integrating 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 Systems via 𝗠𝗖𝗣 👇

If you are building RAG systems and packing many data sources for retrieval, most likely there is some agency present at least at the data source selection for retrieval stage.

This is how MCP enriches the evolution of s in such case (𝘱𝘰𝘪𝘯𝘵 2.):

𝟭. Analysis of the user query: we pass the original user query to a LLM based Agent for analysis. This is where:

➡️ The original query can be rewritten, sometimes multiple times to create either a single or multiple queries to be passed down the pipeline.
➡️ The agent decides if additional data sources are required to answer the query.

𝟮. If additional data is required, the Retrieval step is triggered. We could tap into variety of data types, few examples:

➡️ Real time user data.
➡️ Internal documents that a user might be interested in.
➡️ Data available on the web.
➡️ …

𝗧𝗵𝗶𝘀 𝗶𝘀 𝘄𝗵𝗲𝗿𝗲 𝗠𝗖𝗣 𝗰𝗼𝗺𝗲𝘀 𝗶𝗻:

✅ Each data domain can manage their own MCP Servers. Exposing specific rules of how the data should be used.
✅ Security and compliance can be ensured on the Servel level for each domain.
✅ New data domains can be easily added to the MCP server pool in a standardised way with no Agent rewrite needed enabling decoupled evolution of the system in terms of 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗮𝗹, 𝗘𝗽𝗶𝘀𝗼𝗱𝗶𝗰 𝗮𝗻𝗱 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗠𝗲𝗺𝗼𝗿𝘆.
✅ Platform builders can expose their data in a standardised way to external consumers. Enabling easy access to data on the web.
✅ AI Engineers can continue to focus on the topology of the Agent.

𝟯. Retrieved data is consolidated and Reranked by a more powerful model compared to regular embedder. Data points are significantly narrowed down.
𝟰. If there is no need for additional data, we try to compose the answer (or multiple answers or a set of actions) straight via an LLM.
𝟱. The answer gets analyzed, summarized and evaluated for correctness and relevance:

➡️ If the Agent decides that the answer is good enough, it gets returned to the user.
➡️ If the Agent decides that the answer needs improvement, we try to rewrite the user query and repeat the generation loop.

Are you using MCP in your Agentic RAG systems? Let me know about your experience in the comment section 👇