Running a lower-cost model multiple times can match the vulnerability detection rate of frontier models at a small fraction of the cost.
On a 30-task subset of dfbench, five runs of DeepSeek v4.1 Flash found about 71% of known vulnerabilities, up from about 52% with a single Running it 15 times increased recall further to 80%, roughly 13% of Mythos 5’s cost, which found approximately 69% of the vulnerabilities.
Because LLMs are stochastic, they can find different vulnerabilities each time they analyze the same codebase. Combining findings across runs increases coverage, but can also introduce more false positives.
To control that noise, we used our harness to review and deduplicate findings across runs, then filter them by confidence. In separate validation experiments, this approach generally improves both recall and precision.
These results suggest that repeated sampling can improve the tradeoff between coverage, precision, and cost. The gains depend on the model, the number of runs, and how findings are filtered.
Read the research: depthfirst.com/research/bon-v…
On a 30-task subset of dfbench, five runs of DeepSeek v4.1 Flash found about 71% of known vulnerabilities, up from about 52% with a single Running it 15 times increased recall further to 80%, roughly 13% of Mythos 5’s cost, which found approximately 69% of the vulnerabilities.
Because LLMs are stochastic, they can find different vulnerabilities each time they analyze the same codebase. Combining findings across runs increases coverage, but can also introduce more false positives.
To control that noise, we used our harness to review and deduplicate findings across runs, then filter them by confidence. In separate validation experiments, this approach generally improves both recall and precision.
These results suggest that repeated sampling can improve the tradeoff between coverage, precision, and cost. The gains depend on the model, the number of runs, and how findings are filtered.
Read the research: depthfirst.com/research/bon-v…
5
5
51
6,194
depthfirst retweeted
Almost three years ago this was definitely not consensus. Most folks in research and in security did not agree with me. Looking back at our first vision document, things ended up moving exactly as we wrote. Maybe we got lucky, maybe we predicted correctly, it doesn't matter. What matter is what comes next
When @andreamichi, Daniele, and I started planning on founding @depthfirstlabs, we wrote a document outlining our vision for the future of cybersecurity and our company’s strategy. Today, 2.5 years later, cybersecurity has become one of the most important areas of AI development reread the document and thought it would be interesting to share how our predictions held up and how we have executed on our plan. I’m sharing the text from the original Google Doc we wrote, last modified on June 18th, 2024.
depthfirst.com/post/original-…
Our original thesis was that AI would fundamentally change the economics of vulnerability discovery and exploitation. Finding sophisticated vulnerabilities had historically required scarce expertise and considerable time. We believed AI would dramatically reduce those constraints, giving attackers the ability to find and exploit vulnerabilities at a scale that existing defenses would struggle to keep up with. Today, increasingly capable AI models are discovering vulnerabilities that have gone undetected for years, and powerful open-weight models are making these capabilities accessible to virtually anyone.
One passage from the document feels particularly prescient today: “In a world of abundance intelligence, this situation will not be tenable and will lead to catastrophic consequences if not addressed. The company’s mission is to identify and patch all the vulnerabilities in the software running the world.”
We also believed that AI would make it possible (and necessary) to build defensive systems capable of finding and fixing vulnerabilities at a scale that had never been economically feasible, and set the objective of building “computer-security artificial general intelligence”.
Two years in, we have raised $120m, built a growing team of 70, trained a SOTA cybersecurity model, found vulnerabilities in the biggest open-source projects and major enterprises, and have an eight-figure revenue run rate. AI startups, Fortune 500 companies, and government agencies use depthfirst to reduce risk across their systems and meet today’s security challenges.
We’re growing our team and are looking for excellent, mission driven people across engineering, GTM, and ops. If you’re excited about our mission, consider applying.
2
2
35
49,203
depthfirst retweeted
When @andreamichi, Daniele, and I started planning on founding @depthfirstlabs, we wrote a document outlining our vision for the future of cybersecurity and our company’s strategy. Today, 2.5 years later, cybersecurity has become one of the most important areas of AI development reread the document and thought it would be interesting to share how our predictions held up and how we have executed on our plan. I’m sharing the text from the original Google Doc we wrote, last modified on June 18th, 2024.
depthfirst.com/post/original-…
Our original thesis was that AI would fundamentally change the economics of vulnerability discovery and exploitation. Finding sophisticated vulnerabilities had historically required scarce expertise and considerable time. We believed AI would dramatically reduce those constraints, giving attackers the ability to find and exploit vulnerabilities at a scale that existing defenses would struggle to keep up with. Today, increasingly capable AI models are discovering vulnerabilities that have gone undetected for years, and powerful open-weight models are making these capabilities accessible to virtually anyone.
One passage from the document feels particularly prescient today: “In a world of abundance intelligence, this situation will not be tenable and will lead to catastrophic consequences if not addressed. The company’s mission is to identify and patch all the vulnerabilities in the software running the world.”
We also believed that AI would make it possible (and necessary) to build defensive systems capable of finding and fixing vulnerabilities at a scale that had never been economically feasible, and set the objective of building “computer-security artificial general intelligence”.
Two years in, we have raised $120m, built a growing team of 70, trained a SOTA cybersecurity model, found vulnerabilities in the biggest open-source projects and major enterprises, and have an eight-figure revenue run rate. AI startups, Fortune 500 companies, and government agencies use depthfirst to reduce risk across their systems and meet today’s security challenges.
We’re growing our team and are looking for excellent, mission driven people across engineering, GTM, and ops. If you’re excited about our mission, consider applying.
depthfirst.com/post/original-…
Our original thesis was that AI would fundamentally change the economics of vulnerability discovery and exploitation. Finding sophisticated vulnerabilities had historically required scarce expertise and considerable time. We believed AI would dramatically reduce those constraints, giving attackers the ability to find and exploit vulnerabilities at a scale that existing defenses would struggle to keep up with. Today, increasingly capable AI models are discovering vulnerabilities that have gone undetected for years, and powerful open-weight models are making these capabilities accessible to virtually anyone.
One passage from the document feels particularly prescient today: “In a world of abundance intelligence, this situation will not be tenable and will lead to catastrophic consequences if not addressed. The company’s mission is to identify and patch all the vulnerabilities in the software running the world.”
We also believed that AI would make it possible (and necessary) to build defensive systems capable of finding and fixing vulnerabilities at a scale that had never been economically feasible, and set the objective of building “computer-security artificial general intelligence”.
Two years in, we have raised $120m, built a growing team of 70, trained a SOTA cybersecurity model, found vulnerabilities in the biggest open-source projects and major enterprises, and have an eight-figure revenue run rate. AI startups, Fortune 500 companies, and government agencies use depthfirst to reduce risk across their systems and meet today’s security challenges.
We’re growing our team and are looking for excellent, mission driven people across engineering, GTM, and ops. If you’re excited about our mission, consider applying.
12
6
46
56,380
We are showcasing our work with @nvidia to post-train Nemotron 3.5 Lightning for defensive cybersecurity at GTC.
As AI makes cyberattacks faster, cheaper, and more common, we need to equip defenders with advanced cyber intelligence that is affordable enough to be always on.
ding at depthfirst: powerful, specialized security models that defenders can run on their own infrastructure, with full control over their models and data, at a fraction of today's cost.
We're proud to work with NVIDIA to strengthen the U.S. AI cybersecurity ecosystem and help put sovereign security intelligence in the hands of defenders.
As AI makes cyberattacks faster, cheaper, and more common, we need to equip defenders with advanced cyber intelligence that is affordable enough to be always on.
ding at depthfirst: powerful, specialized security models that defenders can run on their own infrastructure, with full control over their models and data, at a fraction of today's cost.
We're proud to work with NVIDIA to strengthen the U.S. AI cybersecurity ecosystem and help put sovereign security intelligence in the hands of defenders.
0
5
29
16,077
depthfirst retweeted
We have been flagging and blocking malicious packages like litellm for our users but the scale of these attacks is unprecedented and requires a global response.
This is just the tip of the iceberg.
Most of the internet is flooded with vulnerabilities that can now be found cheaply at scale by pretty much anyone.
We have a lot of work to do.
0
3
25
744
depthfirst retweeted
We tested Gemini 4 Argon on @depthfirstlabs dfbench, our benchmark for finding real vulnerabilities in code.
On application and smart-contract tasks, where it rarely refused, it found 74% of known vulnerabilities, the highest recall we have measured. Across all 165 tasks it ly to Mythos 5 (69%{.
On application and smart-contract tasks, where it rarely refused, it found 74% of known vulnerabilities, the highest recall we have measured. Across all 165 tasks it ly to Mythos 5 (69%{.
3
6
41
14,179
depthfirst retweeted
.@depthfirstlabs was named to the Forbes Next Billion-Dollar Startups 2026 list, one of only 20 companies to make it!
Qasim Mithani, Daniele Perito,@andreamichi, and team are securing the world’s software. AI is making it easier than ever to ship software, and easier than ever to attack it.
Forbes closed a $80M Series B led by Meritech Capital less than 90 days after their Series A led by Accel.
Daniele did his postdoc in AI and security at UC Berkeley, was on the founding team at Cash App where he led security, and co-founded Faire. That's the kind of arc Berkeley keeps producing.
The House Fund is proud to be on board for the journey. Congrats to Qasim, Daniele, Andrea, and the entire depthfirst team!
They're hiring across research, engineering, product, and GTM - 15 open roles in San Francisco.
Qasim Mithani, Daniele Perito,@andreamichi, and team are securing the world’s software. AI is making it easier than ever to ship software, and easier than ever to attack it.
Forbes closed a $80M Series B led by Meritech Capital less than 90 days after their Series A led by Accel.
Daniele did his postdoc in AI and security at UC Berkeley, was on the founding team at Cash App where he led security, and co-founded Faire. That's the kind of arc Berkeley keeps producing.
The House Fund is proud to be on board for the journey. Congrats to Qasim, Daniele, Andrea, and the entire depthfirst team!
They're hiring across research, engineering, product, and GTM - 15 open roles in San Francisco.
3
5
14
752
depthfirst retweeted
We evaluated the defensive security capabilities of @OpenAI's GPT 6.1 Sol on @depthfirstlabs dfbench.
On our vulnerability detection tasks, it found about as many vulnerabilities as GPT 5.6 Sol, 56–63% recall against 51–65% across reasoning efforts, at a fraction of the cost: ainst $3.57–$12.15.
It also beat GPT 6 Sol at every effort level, finding more vulnerabilities for less money.
On our vulnerability detection tasks, it found about as many vulnerabilities as GPT 5.6 Sol, 56–63% recall against 51–65% across reasoning efforts, at a fraction of the cost: ainst $3.57–$12.15.
It also beat GPT 6 Sol at every effort level, finding more vulnerabilities for less money.
4
2
28
1,229
depthfirst retweeted
Alignment alone is not enough to make autonomous AI safe.
An agent pursuing a legitimate goal can still reward hack, bypass guardrails, and compromise systems.
We've seen it with the recent OpenAI / Hugging Face and other incidents, where hundreds of agents coordinated with each infrastructure.
The details of those attacks are mind-blowing and show why alignment and security guardrails need to be developed together.
One difficulty in aligning a model is that rewarding the right result doesn’t necessarily teach it the right behavior. It can learn to game the reward instead of doing the intended task.
We’ve seen versions of reward hacking in our own work at depthfirst (see blog for details).
During our research, we’ve seen how security guardrails can support alignment by removing shortcuts that let agents earn rewards without doing the intended task.
After deployment, they constrain unauthorized actions and contain failures.
The more autonomy we give AI, the more we need secure software and infrastructure.
Similar to how we're more comfortable letting our kids run around in an enclosed backyard.
depthfirst.com/post/alignment…
An agent pursuing a legitimate goal can still reward hack, bypass guardrails, and compromise systems.
We've seen it with the recent OpenAI / Hugging Face and other incidents, where hundreds of agents coordinated with each infrastructure.
The details of those attacks are mind-blowing and show why alignment and security guardrails need to be developed together.
One difficulty in aligning a model is that rewarding the right result doesn’t necessarily teach it the right behavior. It can learn to game the reward instead of doing the intended task.
We’ve seen versions of reward hacking in our own work at depthfirst (see blog for details).
During our research, we’ve seen how security guardrails can support alignment by removing shortcuts that let agents earn rewards without doing the intended task.
After deployment, they constrain unauthorized actions and contain failures.
The more autonomy we give AI, the more we need secure software and infrastructure.
Similar to how we're more comfortable letting our kids run around in an enclosed backyard.
depthfirst.com/post/alignment…
0
3
20
680
Today, we’re excited to introduce Agentic Pentesting on the depthfirst platform.
Agentic Pentesting involves autonomous agents that attack your running application like a team of elite security researchers.
Security teams can:
- Find complex vulnerabilities in systems and ith or without source code access
- Confirm whether findings from code scans and other sources can be exploited at runtime
- Run ad hoc agentic pentests or integrate them into CI/CD for new feature releases
- Create tickets in Jira or Linear, or use depthfirst to autonomously remediate vulnerabilities
- Replay an attack after remediation to verify that the vulnerability has been fixed
We built Agentic Pentesting in close collaboration with our customers. We’re grateful to every security and engineering team that helped shape the product and to the depthfirst team that brought it to life.
Learn more from the links in the comments.
Agentic Pentesting involves autonomous agents that attack your running application like a team of elite security researchers.
Security teams can:
- Find complex vulnerabilities in systems and ith or without source code access
- Confirm whether findings from code scans and other sources can be exploited at runtime
- Run ad hoc agentic pentests or integrate them into CI/CD for new feature releases
- Create tickets in Jira or Linear, or use depthfirst to autonomously remediate vulnerabilities
- Replay an attack after remediation to verify that the vulnerability has been fixed
We built Agentic Pentesting in close collaboration with our customers. We’re grateful to every security and engineering team that helped shape the product and to the depthfirst team that brought it to life.
Learn more from the links in the comments.
4
3
54
341,634
We use Codex and other AI models at depthfirst. They’re useful for code review, and we run our Security Reviewer alongside them to see how each model performs when reviewing the same code.
In one internal test repo, Codex raised 3,519 issues (123 security-related). depthfirst es at a >70% true-positive rate.
We wrote about why security review needs deeper application context, how this context informs every pull request review on the depthfirst platform.
Read more in the link in comments.
In one internal test repo, Codex raised 3,519 issues (123 security-related). depthfirst es at a >70% true-positive rate.
We wrote about why security review needs deeper application context, how this context informs every pull request review on the depthfirst platform.
Read more in the link in comments.
7
0
34
2,814
Our researchers built a vulnerability scanner that costs two cents per scan and completes a scan in around two seconds.
The scanner is built with Jev, a classifier that outputs probabilities rather than generating tokens. On dfbench v1, our flagship model, it finds nearly as es as GPT-5.6 Luna, one of the most efficient models by cost per vulnerability.
This new harness breaks vulnerability discovery into parallel tasks, making it possible to scan every package, repository, or commit quickly and at low cost before escalating the hardest cases to other frontier models on the depthfirst platform.
Read more about how we built this in comments 👇
The scanner is built with Jev, a classifier that outputs probabilities rather than generating tokens. On dfbench v1, our flagship model, it finds nearly as es as GPT-5.6 Luna, one of the most efficient models by cost per vulnerability.
This new harness breaks vulnerability discovery into parallel tasks, making it possible to scan every package, repository, or commit quickly and at low cost before escalating the hardest cases to other frontier models on the depthfirst platform.
Read more about how we built this in comments 👇
8
6
77
6,598
We're excited to share our new customer story with @ListenLabs!
Listen Labs’ agentic SDLC enables developers to manage swarms of agents, so security reviews must deliver precise findings scale.
When depthfirst flags a PR issue, it provides context, evidence, and recommended t on the finding before any human decides to merge, keeping security review in lockstep with development.
Thank you Tobias Schindler and the Listen Labs team.
Read the full customer story in the link in comments 👇
Listen Labs’ agentic SDLC enables developers to manage swarms of agents, so security reviews must deliver precise findings scale.
When depthfirst flags a PR issue, it provides context, evidence, and recommended t on the finding before any human decides to merge, keeping security review in lockstep with development.
Thank you Tobias Schindler and the Listen Labs team.
Read the full customer story in the link in comments 👇
1
2
18
640
Read the full customer story here: depthfirst.com/customer-stori…
0
0
3
244
Applications today can pull thousands of third-party packages, and AI agents are accelerating how quickly those dependencies are introduced and changed.
Scanners flag known vulnerabilities across that dependency tree, but security teams still need to know which vulnerable tion can reach.
At depthfirst, we use agents to trace paths through startup commands, frameworks, and transitive dependencies, just like a team of security researchers.
We explore where static analysis misses real execution paths and how we make agentic reachability practical at scale: depthfirst.com/post/reachabil…
Scanners flag known vulnerabilities across that dependency tree, but security teams still need to know which vulnerable tion can reach.
At depthfirst, we use agents to trace paths through startup commands, frameworks, and transitive dependencies, just like a team of security researchers.
We explore where static analysis misses real execution paths and how we make agentic reachability practical at scale: depthfirst.com/post/reachabil…
0
5
22
1,721