Pinned Tweet
1/9 Finally, I can talk about this one.
A few weeks ago, OpenAI told me GPT-6 had basically "cooked" SRE-Bench (our software reverse-engineering benchmark: sre-bench.lol) with a nearly 100% solve rate at pass@4 and ~88% at pass@1; and cheaper!
openai.com/index/gpt-6-as…
A few weeks ago, OpenAI told me GPT-6 had basically "cooked" SRE-Bench (our software reverse-engineering benchmark: sre-bench.lol) with a nearly 100% solve rate at pass@4 and ~88% at pass@1; and cheaper!
openai.com/index/gpt-6-as…
28
45
822
142,855
We have new results for many frontier open-weight models on reverse engineering!
Plot 1 uses rescaled axes for clarity; Plot 2 shows the original scale.
More details: sre-bench.lol/leaderboard
Lots to unpack 🫣! Hope these results help you choose the right models for @reatools!
Plot 1 uses rescaled axes for clarity; Plot 2 shows the original scale.
More details: sre-bench.lol/leaderboard
Lots to unpack 🫣! Hope these results help you choose the right models for @reatools!
1/ As someone who have worked on reverse engineering for 10+ years, I've never seen this much excitement around it. Great work, @morluto and @reatools!
Since RE is having such a moment 🔥, I figured I'd share some updates from sre-bench.lol to help you pick the right model.
1
6
28
1,917
@reatools Thanks @ValsAI for running the eval and sharing the data!
vals.ai/benchmarks/sre… has the updated results!
vals.ai/benchmarks/sre… has the updated results!
0
0
1
208
Oh, shoot. I forget to mention sre-bench.lol is our reverse engineering benchmark 😂
1/ As someone who have worked on reverse engineering for 10+ years, I've never seen this much excitement around it. Great work, @morluto and @reatools!
Since RE is having such a moment 🔥, I figured I'd share some updates from sre-bench.lol to help you pick the right model.
1
1
7
790
1/ As someone who have worked on reverse engineering for 10+ years, I've never seen this much excitement around it. Great work, @morluto and @reatools!
Since RE is having such a moment 🔥, I figured I'd share some updates from sre-bench.lol to help you pick the right model.
Since RE is having such a moment 🔥, I figured I'd share some updates from sre-bench.lol to help you pick the right model.
5
12
78
10,488
7/ More results are on the way, including Gemini 4 evaluated without cyber-specific refusals.
Very curious to see how it stacks up against the current leaders.
Very curious to see how it stacks up against the current leaders.
1
0
3
281
8/ And once again, a huge thanks to all my collaborators for making agentic RE something we can actually measure, track, and better understand! @f3ixiao @nikilravi @xiangyuqi_pton @Eric_Wallace_ @ralucaadapopa @GuannanWei @RobinDing3, and Jeremy, Nicholas, Jinhao, and Matthew (lol, my fault but I couldn't find your X handles 😂)!
0
0
3
271
🚀 What if you could vibe-code ANY existing app, even without its source code?
🎬 Check out our demo! We added brand-new AI functionality directly (and reliably!) to Apple's Preview app, without any source code!
Just Astra Max modifying the existing app in under 20 min, using less than 10% of the weekly token allowance on a $200 subscription!
We've cept skill repository to help agents directly modify existing applications:
🔗 github.com/Pasta-Lab/edit…
And I think this points to something MUCH bigger.
I've never seen reverse engineering become such a hot topic in the past 10 years! 🔥 But with AI agents becoming incredibly capable at understanding binaries, I think it's time to ask a bigger question:
What is the future of reverse engineering?
Here's our answer: Omniform Software Engineering (OmniSE).
Over the past few months, following our SRE-Bench work, my research group has been exploring this new direction.
SRE-Bench showed how capable AI agents have become at understanding and reverse-engineering binaries.
But why should we stop at understanding software? Why not let agents directly develop on top of it?
That's the vision behind OmniSE: Software engineering should no longer have to start with source code. It should be possible to start with software in ANY form! Whether you have source code, compiled binaries, or an existing application, you should be able to directly customize and extend it.
Think about a simple scenario.
You love an app, but wish it had one additional feature.
Traditionally, you have to wait for the original developers to implement it.
Today, with increasingly powerful AI agents, you might instead reverse-engineer and rebuild the entire application.
But think about how many tokens it would take to recreate something as mature as Apple Preview! Even worse, rebuilding everything can easily miss subtle but critical functionality. You don't want to spend your time repeatedly fixing an app that already worked perfectly well!
But why rebuild an entire application just to add ONE feature?
With OmniSE, you can bring your AI agent directly to the existing app. Let it understand the software, modify what is needed, and preserve the rest.
This is what excites me most:
Reverse engineering is no longer just a cybersecurity capability. It is becoming a fundamental software development capability that everyone can use and benefit from.
And I believe this is much more than a new task for AI agents. It could represent an entirely new paradigm of software engineering.
Imagine a future where you don't have to adapt yourself to the software you use. Instead, your AI agent adapts the software to YOU.
Every app could become your own customized app.
That's the future of software we want to build. 🚀 #rea #reverseengineeering #openai #codex #deepmind
@ColumbiaCompSci @Columbia @DAP__Lab
0
2
10
1,020
🎬 Check out our demo! We added brand-new AI functionality directly (and reliably!) to Apple's Preview app, without any source code!
Just Astra Max modifying the existing app in under 20 min, using less than 10% of the weekly token allowance on a $200 subscription!
We've cept skill repository to help agents directly modify existing applications:
🔗 github.com/Pasta-Lab/edit…
And I think this points to something MUCH bigger.
I've never seen reverse engineering become such a hot topic in the past 10 years! 🔥 But with AI agents becoming incredibly capable at understanding binaries, I think it's time to ask a bigger question:
What is the future of reverse engineering?
Here's our answer: Omniform Software Engineering (OmniSE).
Over the past few months, following our SRE-Bench work, my research group has been exploring this new direction.
SRE-Bench showed how capable AI agents have become at understanding and reverse-engineering binaries.
But why should we stop at understanding software? Why not let agents directly develop on top of it?
That's the vision behind OmniSE: Software engineering should no longer have to start with source code. It should be possible to start with software in ANY form! Whether you have source code, compiled binaries, or an existing application, you should be able to directly customize and extend it.
Think about a simple scenario.
You love an app, but wish it had one additional feature.
Traditionally, you have to wait for the original developers to implement it.
Today, with increasingly powerful AI agents, you might instead reverse-engineer and rebuild the entire application.
But think about how many tokens it would take to recreate something as mature as Apple Preview! Even worse, rebuilding everything can easily miss subtle but critical functionality. You don't want to spend your time repeatedly fixing an app that already worked perfectly well!
But why rebuild an entire application just to add ONE feature?
With OmniSE, you can bring your AI agent directly to the existing app. Let it understand the software, modify what is needed, and preserve the rest.
This is what excites me most:
Reverse engineering is no longer just a cybersecurity capability. It is becoming a fundamental software development capability that everyone can use and benefit from.
And I believe this is much more than a new task for AI agents. It could represent an entirely new paradigm of software engineering.
Imagine a future where you don't have to adapt yourself to the software you use. Instead, your AI agent adapts the software to YOU.
Every app could become your own customized app.
That's the future of software we want to build. 🚀 #rea #reverseengineeering #openai #codex #deepmind
@ColumbiaCompSci @Columbia @DAP__Lab
Just Astra Max modifying the existing app in under 20 min, using less than 10% of the weekly token allowance on a $200 subscription!
We've cept skill repository to help agents directly modify existing applications:
🔗 github.com/Pasta-Lab/edit…
And I think this points to something MUCH bigger.
I've never seen reverse engineering become such a hot topic in the past 10 years! 🔥 But with AI agents becoming incredibly capable at understanding binaries, I think it's time to ask a bigger question:
What is the future of reverse engineering?
Here's our answer: Omniform Software Engineering (OmniSE).
Over the past few months, following our SRE-Bench work, my research group has been exploring this new direction.
SRE-Bench showed how capable AI agents have become at understanding and reverse-engineering binaries.
But why should we stop at understanding software? Why not let agents directly develop on top of it?
That's the vision behind OmniSE: Software engineering should no longer have to start with source code. It should be possible to start with software in ANY form! Whether you have source code, compiled binaries, or an existing application, you should be able to directly customize and extend it.
Think about a simple scenario.
You love an app, but wish it had one additional feature.
Traditionally, you have to wait for the original developers to implement it.
Today, with increasingly powerful AI agents, you might instead reverse-engineer and rebuild the entire application.
But think about how many tokens it would take to recreate something as mature as Apple Preview! Even worse, rebuilding everything can easily miss subtle but critical functionality. You don't want to spend your time repeatedly fixing an app that already worked perfectly well!
But why rebuild an entire application just to add ONE feature?
With OmniSE, you can bring your AI agent directly to the existing app. Let it understand the software, modify what is needed, and preserve the rest.
This is what excites me most:
Reverse engineering is no longer just a cybersecurity capability. It is becoming a fundamental software development capability that everyone can use and benefit from.
And I believe this is much more than a new task for AI agents. It could represent an entirely new paradigm of software engineering.
Imagine a future where you don't have to adapt yourself to the software you use. Instead, your AI agent adapts the software to YOU.
Every app could become your own customized app.
That's the future of software we want to build. 🚀 #rea #reverseengineeering #openai #codex #deepmind
@ColumbiaCompSci @Columbia @DAP__Lab
5
5
28
30,330
🎬 If the demo video looks blurry due to X's compression, you can watch the full-quality version on YouTube!
👉 youtube.com/watch?v=TyI4MO…
👉 youtube.com/watch?v=TyI4MO…
1
0
0
264
This exploration has been driven by my students Jeffery Tian and Jeremy Spence, with valuable discussions with my colleagues @RobinDing3 and @GuannanWei at Pasta Lab.
It has also benefited from continued conversations with @xiangyuqi_pton (OpenAI) and @f3ixiao (Google DeepMind), building on our earlier SRE-Bench collaboration.
Excited to keep pushing this direction together!
It has also benefited from continued conversations with @xiangyuqi_pton (OpenAI) and @f3ixiao (Google DeepMind), building on our earlier SRE-Bench collaboration.
Excited to keep pushing this direction together!
0
0
0
152
1/3 🧵
Collaborating with the @CrackmesOne community on new AI reverse-engineering benchmarks has been a blast!
In just a few days, we tested two fascinating challenges featuring build-time metamorphism, a gate-level custom CPU, encrypted memory, and SIGSEGV-driven decryption and control-flow obfuscation.
They look brutally fun to reverse manually. Fortunately (and arguably unfortunately), GPT-6 solved both in under 20 minutes each ;p
Collaborating with the @CrackmesOne community on new AI reverse-engineering benchmarks has been a blast!
In just a few days, we tested two fascinating challenges featuring build-time metamorphism, a gate-level custom CPU, encrypted memory, and SIGSEGV-driven decryption and control-flow obfuscation.
They look brutally fun to reverse manually. Fortunately (and arguably unfortunately), GPT-6 solved both in under 20 minutes each ;p
6
1
19
4,489
2/3
we tested crackmes.one/crackme/6aa471… from github.com/victormeloasm/; and crackmes.one/crackme/692c1d… from xutaxkamay.com
feel free to crack them yourself!
we tested crackmes.one/crackme/6aa471… from github.com/victormeloasm/; and crackmes.one/crackme/692c1d… from xutaxkamay.com
feel free to crack them yourself!
1
0
4
433