Observability and Evals Masterclass x.com/i/broadcasts/1…
1
1
11
316
Alex Booker retweeted
Every AI chat is built for one human

So I put an AI in a group chat with 3 people. It knows who's away, who can't do Mondays, and who gets to make the call

Multiplayer agents, built with @mastra 👇
0
4
7
430
Alex Booker retweeted
Mastra observability now has interactive histograms. See patterns and find errors faster.
1
2
13
1,594
What happens after you find an agent failure?

On Thursday, we'll show you what should happen - turn failed traces into datasets, compare changes with experiments and scorers, and automate regression evals

Join us as we build this eval loop live!

luma.com/mastra-18cw?tk…
0
0
7
341
Why not?

Send it to your factory. See what it comes up with. Go from there!
I'm now building an "improvements" feature into all of my personal vibecoded applications: a freeform text input that's hooked up to an agent that yolo implements whatever is submitted.

It's an interesting experience -- the activation energy barrier to sanding any rough edge or ature becomes extremely low. Someone else who uses one of the applications I created has already implemented 150 improvements over the course of a few days. This pattern feels like a new kind of tool -- liquid software? -- and every application that doesn't have this feature now has a slightly rigid and anachronistic flavor. Why can't I just tell it how I want it to be different?

Now, it's not obvious to me how this pattern can or should scale. Does it become unmaintainable after a while? (Maybe model intelligence improvement outstrips my feature appetite such that it's fine?) How does collaborative software work in this model? Can software designed this way be secure and reliable?

Whatever the larger questions about broader deployment, it definitely works great in a personal context. It reminds me a little of the visions of Lisp/Lisp machines, Smalltalk, and HyperCard -- systems that the user can take control of and modify. Those visions never quite worked, though: the vast majority of software today is not end-user modifiable in any useful way. So, while this "liquid" affordance is a very small touch, the way it changes one's relation to the software feels a bit transgressive and empowering in a Lickliderian and Engelbartian way.
0
0
0
305
Been a rough month. Sorry to say it's going to get worse before it gets better. 😬
October in a nutshell:

- PHP got hacked
- Railway got hacked
- OpenCode got hacked
- Binance got hacked
- Polymarket got hacked
- GitHub Actions got hacked
- SoftBank got hacked
- Cursor marketplace got hacked
- Coinbase vendor got hacked
- Fireworks AI got hacked
- LiteLLM got hacked
- Capcom got hacked
- npm packages got hacked
- TiDB Cloud got hacked
- Warp's vendor got hacked

happy cybersecurity awareness month
0
0
3
947
Fair play! This is one of the coolest little demos I've seen lately. Just spent half an hour poking around my favourite spots in London. Very charming.
introducing tiny.london/world

after tiny.london & isometric.nyc went viral, dozens people did same maps for their cities. i put all of them in one place

from tashkent to goiânia check how similar but different they are!
0
0
1
564
Slack isn't going anywhere soon.

The agent experience isn't great right now, and progress feels slow. But it's not enough to build a better mousetrap. Slack is too hard to leave. Your message history and connections live there, and the network effects are huge.

If something does dethrone Slack, it'll be less distracting, help you deal with fewer messages less often, and feel agent native.
If it happens, it'll take a while. And it'll come from a player that already has distribution -
like Linear or Notion 😉
prediction: for the first time ever, slack will see major competition

it will happen in the next few months

by end of 2027, majority of modern silicon valley companies will be using something else
1
0
5
657
Alex Booker retweeted
Every vector database has its own SDK, query syntax, and filter rules.

Over the last six months we've added seven: same 𝚌𝚛𝚎𝚊𝚝𝚎𝙸𝚗𝚍𝚎𝚡, 𝚞𝚙𝚜𝚎𝚛𝚝, 𝚚𝚞𝚎𝚛𝚢, and 𝚏𝚒𝚕𝚝𝚎𝚛 on Weaviate, Couchbase, OpenSearch, Turbopuffer, etc

Swap stores w/o rewriting your code:
12
5
60
5,223
We're making the Mastra Observability dashboard best-in-class

And everything in it works headless too

So you can ask your coding agent to inspect traces and aggregates, and even fix things - potentially automatically 👀
3
0
8
388
Two latency metrics every agent should track

TTFT → how fast your model responds

TTFC → how long your user actually waits

Fast TTFT but slow TTFC? You probably introduced slow context injection or some other processor
2
0
8
624
Replying to @bookercodes
This is on my mind because we just added both to Mastra's Traces table

We're adding it to dashboards (follow trends) and experiments next
0
0
0
105
I've spent the last 2 months going deep on observability and evals

In next week's workshop I'll skip the cruft and teach you everything I know
1
0
3
327
Right now Mastra traces only supports two statuses. Ok or Error

We're adding Suspended, Aborted, Limited, Blocked and more

More granular means you can tell at a glance why a run stopped!
1
1
5
362
Observational Memory comes out on top. Again 😏

Here's how it works
Agent memory gets harder the longer a chat runs.

We wondered: how well does Observational Memory hold up at 100K, 1M and 10M tokens? So we ran it on BEAM, the long-term memory benchmark, with default settings.

Turns out it has the top published scores at 100K and 1M:
1
4
10
541
On a mission to build the cleanest observability dash for Mastra agents

Today we added a histogram

And input previews - so you see a clean input and not a gross blob of JSON
1
2
13
489
Replying to @bookercodes
Histogram is @mfrachet and @ShlomesLew's handywork
0
0
0
95
Testing adding a marker for each deployment so you can correlate agent metrics with deploys

Would this be helpful?
1
0
5
295
Mastra Observability is starting to look very comprehensive indeed 🤓☝️
Today we're excited to announce span queries and span groups in Mastra!

We started with trace-level queries; now we're diving to span-level which is more granular AND more performant.

Filter/group by time range, span type, name, status, error, duration:
0
2
13
598