Jackson Clark

@HacksonClark

Oct 15

Cognition

@cognition

Oct 15

Introducing Devin Oncall: Devin automatically investigates alerts and bugs, helping your team diagnose and resolve production incidents.

Devin Oncall already handles thousands of alerts every day for some of the world’s largest companies. Let it become your SRE today.

Jackson Clark

@HacksonClark

Oct 27

Logan Kilpatrick

@OfficialLoganK

Oct 15

We are hosting our first "Bench Summit" to bring together people excited about pushing the frontier of benchmarking, model evals, and more.

Great benchmarks are a gating function for model progress, lots to do here!

Join us in SF: rsvp.withgoogle.com/events/bench-s…

Ratul Mahajan

@ratulm

Oct 23

Ratul Mahajan

@ratulm

Mar 19

Early report from the first project we are pursuing as part of the "self-defining systems" research, led by @sudopowr.

So much potential to lower operational toil and MTTR!

Ofir Press

@OfirPress

Oct 22

Kilian Lieret

@KLieret

Oct 22

Let's fix bugs before they cause a global crisis. Updated SWE-sweep leaderboard sees OpenAI and Anthropic models double the number of bugs found, while halving costs. Astra is the best model, but still finds less than 10% of all real-world bugs tested. Details in 🧵

John Yang

@jyangballin

Oct 22

Kilian Lieret

@KLieret

Oct 22

Let's fix bugs before they cause a global crisis. Updated SWE-sweep leaderboard sees OpenAI and Anthropic models double the number of bugs found, while halving costs. Astra is the best model, but still finds less than 10% of all real-world bugs tested. Details in 🧵

Tianyin Xu

@tianyin_xu

Oct 21

Pial@SaadMRP

Oct 21

1/8

In our latest experiment, we let Jev, a decision model, investigate system failures and asked an LLM to check its diagnosis.

Across 21 SREGym-Lite faults, review raised Jev’s pass rate from 78.1% to 89.5%, with diagnoses averaging 38 seconds.

Josh McGrath

@j_mcgraph

Oct 21

Alex Ratner

@ajratner

Oct 20

The @SnorkelAI team did an audit of these findings using our internal QC platform.

Tl/dr: We agree with many of the findings, but ~55% of the fixes from @preferencemodel had their own issues with reward hacking or over-prescriptiveness.

Additionally: ~40% of the issues found were already tracked in the upstream repo - including 21 tasks with outstanding fix PRs!

Building datasets like this - and maintaining ambitious open benchmarks around them - is hard. We believe it requires open, collaborative review with community maintainers and relevant domain experts to get right.

@terminalbench was one of the first community efforts to build an open, continuous benchmark, and drive a vision of how OSS contributes to the rapidly advancing frontier of AI. We hope that the community will invest in working collaboratively with benchmark maintainers and task authors to keep this vision alive.

---

Some takeaways + details of our preliminary audit:

(1) We confirm 100% of the reward hacking / verification gap findings using our QC system. However, we find that 17/44 (39%) of the corrected tasks are just as or more reward hackable. Additionally, we don’t yet see evidence that these verification gaps meaningfully affected benchmark rollouts within CIs, though our analysis here is still ongoing.

(2) Our preliminary review raises questions about the underspecification findings for 22/41 (54%) of the flagged tasks, including 7 explicitly marked by task authors as intentional design choices. These cases warrant expert peer review and adjudication: balancing over/under-specification is a critical and tricky design decision - much of human intelligence concerns dealing correctly with reasonable and realistic underspecification - and we believe these corrections made to the tasks are likely too over-specified, reducing realism and difficulty relative to spec.

(3) Around 40% of the issues found were already tracked in the upstream repo (including 21 tasks with outstanding fix PRs)! OSS projects require community efforts to correct identified open issues, and ideally audits like these are done collaboratively with task authors and maintainers directly, in the traditional community spirit that has kept OSS alive for decades.

Jackson Clark

@HacksonClark

Oct 21

Pial@SaadMRP

Oct 21

1/8

In our latest experiment, we let Jev, a decision model, investigate system failures and asked an LLM to check its diagnosis.

Across 21 SREGym-Lite faults, review raised Jev’s pass rate from 78.1% to 89.5%, with diagnoses averaging 38 seconds.

Jackson Clark

@HacksonClark

Oct 17

Nathan LeClaire

@dotpem

Oct 17

SREGym's blog post on using Jev to troubleshoot incidents is a perfect example of intelligent graph traversal. A lot of debugging an incident is just following the bread crumb trail.

sregym.com/blog/jev-drive…

Tianyin Xu

@tianyin_xu

Oct 15

Jackson Clark

@HacksonClark

Oct 15

Huge congrats to @cognition for taking the #1 spot on #SREGym! 🎉

It’s been incredible to have a front-row seat to the progress in AI SRE. Shoutout to @anhangz, @yunpark93, and @jake_rachleff for an amazing release pushing the frontier of autonomous incident response.

One step humans don't have to be on call. 🚀

SREGym 1.1 is out now, with 2.0 on the way to push the frontier even further.

📊 Follow the progress: sregym.com
📝 More on Devin Oncall: devin.ai/blog/introduci…

Tianyin Xu

@tianyin_xu

Oct 15

Ben Cohen

@bthecohen

Oct 15

@datadoghqhat Scout does not just fit our harness, we also ran the public SREGym benchmark: 90 faults, with the supplied Stratus harness.

Scout 1.0 xhigh resolves 80 of 90 faults, competitive with the best frontier models at least 2× cheaper. Scout 1.0 medium beats GLM-5.3 at more than 4× lower cost.

Tianyin Xu

@tianyin_xu

Oct 15

Ben Cohen

@bthecohen

Oct 15

@datadoghqhat Scout does not just fit our harness, we also ran the public SREGym benchmark: 90 faults, with the supplied Stratus harness.

Scout 1.0 xhigh resolves 80 of 90 faults, competitive with the best frontier models at least 2× cheaper. Scout 1.0 medium beats GLM-5.3 at more than 4× lower cost.

Jackson Clark

@HacksonClark

Oct 15

Ben Cohen

@bthecohen

Oct 15

Introducing Scout 1.0, @datadoghq's first model post-trained for investigating production incidents. Scout matches frontier-model investigation quality at a fraction of the cost, which makes it practical to investigate more issues, earlier.

We trained Scout on realistic real-world incidents and production telemetry, using reinforcement learning to reward accurate root causes backed by evidence.

Jackson Clark

@HacksonClark

Oct 14

Bilgin Ibryam

@bibryam

Oct 13

Jev-Driven SRE Diagnosis:
What Worked and What Failed
sregym.com/blog/jev-drive…

Jeff Ma

@18jeffreyma

Oct 14

Dilara Soylu

@dilarafsoylu

Oct 14

AI is cracking math problems. What's holding it back in engineering?

The cost of evaluation. Agents can generate ideas in seconds, but testing them can take hours or days.

💡 Meet DuoDiscover: where evaluation is part of what agents discover 👇🧵

Some highlights:
🏆 #1 on 🐍 2× the experts on SWE-fficiency
🚀 1.92× on vLLM; 1.44× on FreeToken
🔥 2.68× over production TPU kernels
🖥️ Beats the best human and AI-designed CPU prefetchers

Jackson Clark

@HacksonClark

Oct 13

Tianyin Xu

@tianyin_xu

Oct 13

Last Thursday, I went to the Frontier Data Summit with @HacksonClark to present #SREGym. It was fun. We wrote down our experience and our next step (SREGym 2.0) which we shared at the summit. Thanks @SnorkelAI for having us, especially @_Incynthia, Francesca, @bradenjhancock and @ajratner.

Pushing the AI frontier is like climbing mountains. We're always excited to aim for the next peak. It's great to do it with the open community #SREGym has been building. We'll make it!

See our blog post,
sregym.com/blog/frontier-…

(Thanks @SaadMRP for editing it)