Pinned Tweet
Introducing Scorers: Agents that grade your agents.
Use LLM-as-a-judge to grade past coding agent sessions on:
• Quality
• Efficiency
• Compliance
• Or any custom dimension
Scores feed into performance measurements and automatic self-improvement for software factories
Use LLM-as-a-judge to grade past coding agent sessions on:
• Quality
• Efficiency
• Compliance
• Or any custom dimension
Scores feed into performance measurements and automatic self-improvement for software factories
19
14
207
580,098
If you have 100 agent PRs waiting for your review, this should help you get through them:
We built a skill for addressing the code review bottleneck: /merge-risk
This assesses the risk of any PR across 5 criteria: alignment, impact, verification, rollback cost, and error monitoring.
1. Alignment: Does the code follow the issue and PR description, or are there tended side effects?
2. Impact: What is the blast radius of the change, and is it touching sensitive code paths like auth, permissions, or billing?
3. Verification: Is the code well verified, both with test coverage and end-to-end computer use artifacts?
4. Rollback cost: How easily can this be rolled back, and what is the rollback procedure in the case of an incident? Think one-way vs. two-way doors.
5. Error detection: If something goes wrong in production, how much visibility will we have? Are errors swallowed, logged, or reported to an error monitoring service?
You can run it as a PR comment automation, or make it a pre-PR check for the agent to derisk its own work before tagging in a human.
2
1
21
3,402
Warp retweeted
Inspired by @poteto I benchmarked Warp's factory to see if adversarial code review was worth it.
On the task set I checked it increased cost by 140% while moving quality up by 7.5%. So, probably not, although maybe for really impt changes (many caveats here, you should watch I still believe though that benchmarking is 100% the right approach and everyone should try to set up a factory where they can measure rather than go on vibes.
On the task set I checked it increased cost by 140% while moving quality up by 7.5%. So, probably not, although maybe for really impt changes (many caveats here, you should watch I still believe though that benchmarking is 100% the right approach and everyone should try to set up a factory where they can measure rather than go on vibes.
63
18
536
45,576
Warp retweeted
It's worth building your own verification tools for agents: code review loops, e2e simulations, and merge risk checks.
I'll be hosting a live session next week to build these together!
Tue, Nov 10th at 2pm ET
RSVP: luma.com/warp-daxp
I'll be hosting a live session next week to build these together!
Tue, Nov 10th at 2pm ET
RSVP: luma.com/warp-daxp
Alright, it's time to address the code review bottleneck.
Here's 3 steps worth adding to your agent workflows:
1. Build your own code review loop
2. Verify flows end-to-end with simulated environments
3. Score and mitigate risk with /merge-risk x.com/i/article/2118…
8
5
49
8,224
If you're buried under 1000 agent code review requests, read this:
Alright, it's time to address the code review bottleneck.
Here's 3 steps worth adding to your agent workflows:
1. Build your own code review loop
2. Verify flows end-to-end with simulated environments
3. Score and mitigate risk with /merge-risk x.com/i/article/2118…
5
1
35
9,176
If you're building a software factory, read this first:
I often see tweets that a software factory can be built with a string of prompts tying together cloud agents. My experience is that this is false.
Companies often start there and regret it because it doesn’t scale. x.com/i/article/2117…
3
1
74
16,001
Warp retweeted
Your team should have a shared configuration for your agents that lives in code. The same way you use Terraform to deploy cloud infrastructure, you use a factory configuration to deploy your software factory.
This gives you the typical advantages of version controlled systems: check in changes with code owners, test changes with rollback support, and review updates as diffs.
This gives you the typical advantages of version controlled systems: check in changes with code owners, test changes with rollback support, and review updates as diffs.
25
7
67
6,554
Warp retweeted
We've landed on 7 core components to building a good software factory. Here's the full overview:
A lot more goes into building a software factory than you might think. There are 7 must-haves we've landed on:
- Configured as code
- Central conversation history
- Benchmarking tools
- Factory chat
- Artifacts
- Computer use
- Agent debugger
In detail: x.com/i/article/2081…
3
12
216
27,105
Warp retweeted
There are many ways to define “software factory”, and here’s the one we’ve settled on: centralizing your system of agents into one place to build software. This means going from custom coding setups on everyone's machines to a single definition in code that runs in the cloud. The same way engineers agree on CI, we should agree on how agents are run.
Why is this important? Better security management, better ROI tracking, and mainly, improving agent efficiency as a team – instead of copying config to everyone’s local setups and trying to manage individual skill sprawl.
Why is this important? Better security management, better ROI tracking, and mainly, improving agent efficiency as a team – instead of copying config to everyone’s local setups and trying to manage individual skill sprawl.
18
9
125
78,812
If you're planning to build a software factory, read this first:
A lot more goes into building a software factory than you might think. There are 7 must-haves we've landed on:
- Configured as code
- Central conversation history
- Benchmarking tools
- Factory chat
- Artifacts
- Computer use
- Agent debugger
In detail: x.com/i/article/2081…
1
2
25
7,300
We've been running benchmarks on a variety of tasks at Warp. For some, it turns out that open weights models are actually more expensive than their frontier equivalents.
Run benchmarks on your own data!
Run benchmarks on your own data!
3d game demos aren't the best way to judge a model's coding ability according to @theo. neither is SWE-bench, CursorBench, or TermBench.
instead, you should benchmark models against your own traces to find the best one for your tasks.
here's how the Warp team benchmarks various which helped reduce cost / PR from $80 to $20 in our software factory (alongside uncovering a number of unintuitive findings!)
Demo:
2
0
3
3,463
With software factories, our team has flipped from mainly terminal sessions to inviting agents into our workspaces:
- Slack / MS Teams
- GitHub PRs
- Linear / JIRA / Azure DevOps
The reason? Human attention. Here's Warp engineer Lili showing how we hop between tools today:
- Slack / MS Teams
- GitHub PRs
- Linear / JIRA / Azure DevOps
The reason? Human attention. Here's Warp engineer Lili showing how we hop between tools today:
9
4
79
875,141
Reposting from our CEO:
On Friday, we became aware of what we believe to be an active and narrow-banded exploitation of a vendor in Warp's infrastructure. We detected the operation within hours and were able to remediate and mitigate the impact. We have been in touch with the vendor about the vulnerability.
Everyone who uses Warp places a lot of trust in us and that’s not something we take lightly. I’m very sorry that this incident has occurred.
The attack appears to have come from an agent swarm using a very sophisticated penetration and escalation technique, likely starting with a hole in a vendor dependency, and ultimately accessing a limited amount of agent trace and telemetry data.
We have been working over the weekend and we have notified anyone who we have positively identified as impacted. As of today, we believe an upper bound of roughly 1% of Warp users may have been impacted.
We are working closely with the team at Crowdstrike to do a more extensive and comprehensive investigation and have engaged law enforcement.
Anyone who has been impacted will receive instructions from us on action they need to take. We will continue to post updates as we learn more.
1
1
14
7,147
Warp retweeted
On Friday, we became aware of what we believe to be an active and narrow-banded exploitation of a vendor in Warp's infrastructure. We detected the operation within hours and were able to remediate and mitigate the impact. We have been in touch with the vendor about the vulnerability.
Everyone who uses Warp places a lot of trust in us and that’s not something we take lightly. I’m very sorry that this incident has occurred.
The attack appears to have come from an agent swarm using a very sophisticated penetration and escalation technique, likely starting with a hole in a vendor dependency, and ultimately accessing a limited amount of agent trace and telemetry data.
We have been working over the weekend and we have notified anyone who we have positively identified as impacted. As of today, we believe an upper bound of roughly 1% of Warp users may have been impacted.
We are working closely with the team at Crowdstrike to do a more extensive and comprehensive investigation and have engaged law enforcement.
Anyone who has been impacted will receive instructions from us on action they need to take. We will continue to post updates as we learn more.
Everyone who uses Warp places a lot of trust in us and that’s not something we take lightly. I’m very sorry that this incident has occurred.
The attack appears to have come from an agent swarm using a very sophisticated penetration and escalation technique, likely starting with a hole in a vendor dependency, and ultimately accessing a limited amount of agent trace and telemetry data.
We have been working over the weekend and we have notified anyone who we have positively identified as impacted. As of today, we believe an upper bound of roughly 1% of Warp users may have been impacted.
We are working closely with the team at Crowdstrike to do a more extensive and comprehensive investigation and have engaged law enforcement.
Anyone who has been impacted will receive instructions from us on action they need to take. We will continue to post updates as we learn more.
22
22
194
58,175
Warp retweeted
Want better quality output from your agent? Give it a way to verify its own work.
Love what Warp is doing here: every agent gets a computer and brings back a video of what it did. I do something similar in my workflow - on top of E2E tests, Claude gives me recorded videos + screenshots as part of the evidence work is "done".
My bet is verification becomes one of the most important problems to work on this year.
Love what Warp is doing here: every agent gets a computer and brings back a video of what it did. I do something similar in my workflow - on top of E2E tests, Claude gives me recorded videos + screenshots as part of the evidence work is "done".
My bet is verification becomes one of the most important problems to work on this year.
verification is the biggest bottleneck for coding agents right now according to @addyosmani.
we @warpdotdev have put a lot of thought into building a good computer-use tool for Warp Factories. every agents gets a computer to navigate the apps they built, take screenshots and em back for human review.
3-4mins of activity get edited down to 15 clips stitched together with cursor clicks, mouse animations, and key presses.
we've been able to dramatically cut our time to merge and unlocked faster iteration + review cycles, especially for projects that are hard to mock out locally!
99
53
671
73,177
Warp retweeted
Stop asking agents for text summaries of a PR. They're capable of so much more:
- Architecture diagrams to visually explain the change. Could be ascii, mermaid, HTML, or animated walkthroughs. /show-me is a great starting point.
- Screen recordings verifying the change ses can do this now with Chrome DevTools or computer use. Warp Factories will auto-edit video and highlight cursor clicks as well.
- Planning documents when humans were in the loop, checked in as code. This is optional, but a product spec explaining the purpose and expected user flow will a) guide the agent to better screen recording verification, and b) help the reviewer when looking at another engineer's work.
Warp Factories shows all artifacts connected to an agent session to make these easy to find.
- Architecture diagrams to visually explain the change. Could be ascii, mermaid, HTML, or animated walkthroughs. /show-me is a great starting point.
- Screen recordings verifying the change ses can do this now with Chrome DevTools or computer use. Warp Factories will auto-edit video and highlight cursor clicks as well.
- Planning documents when humans were in the loop, checked in as code. This is optional, but a product spec explaining the purpose and expected user flow will a) guide the agent to better screen recording verification, and b) help the reviewer when looking at another engineer's work.
Warp Factories shows all artifacts connected to an agent session to make these easy to find.
34
15
355
53,146
If an agent's PR doesn't have a screenshot verification, it's not worth reviewing. Add this to your PR requirements:
verification is the biggest bottleneck for coding agents right now according to @addyosmani.
we @warpdotdev have put a lot of thought into building a good computer-use tool for Warp Factories. every agents gets a computer to navigate the apps they built, take screenshots and em back for human review.
3-4mins of activity get edited down to 15 clips stitched together with cursor clicks, mouse animations, and key presses.
we've been able to dramatically cut our time to merge and unlocked faster iteration + review cycles, especially for projects that are hard to mock out locally!
3
1
20
3,744
Warp retweeted
verification is the biggest bottleneck for coding agents right now according to @addyosmani.
we @warpdotdev have put a lot of thought into building a good computer-use tool for Warp Factories. every agents gets a computer to navigate the apps they built, take screenshots and em back for human review.
3-4mins of activity get edited down to 15 clips stitched together with cursor clicks, mouse animations, and key presses.
we've been able to dramatically cut our time to merge and unlocked faster iteration + review cycles, especially for projects that are hard to mock out locally!
we @warpdotdev have put a lot of thought into building a good computer-use tool for Warp Factories. every agents gets a computer to navigate the apps they built, take screenshots and em back for human review.
3-4mins of activity get edited down to 15 clips stitched together with cursor clicks, mouse animations, and key presses.
we've been able to dramatically cut our time to merge and unlocked faster iteration + review cycles, especially for projects that are hard to mock out locally!
24
11
220
89,738
If you're setting up a software factory, use this configuration format:
Software factories should be configured as code. With Warp, your factory lives in a repo as markdown and YAML files, so your agents, automations, runners, benchmarks, and even custom model routers are all in one place.
Here, I walk through a typical Warp factory config and show ion for self improvement!
It's pretty awesome DX.
0
0
8
2,586