Best Paper Award to our ICML Workshop paper 'Fantastic Adaptive Taxonomies and How to Use Them'.
Let me summarize briefly what the paper is about: Failure taxonomies are becoming increasingly important.
We show that failure taxonomies can be used in multiple ways: 1. As a for best-of-N judges, 2) as a mutation feedback mechanism in optimization loops and 3) as runtime feedback for coding agents.
Previously people used MAST and other hand-made fixed failure taxonomies. In this paper we show how to create the failure taxonomy adaptively and dynamically: We observe agent rollouts and create an adaptive failure taxonomy, bespoke to the agent weaknesses and task challenges. These adaptive taxonomies give a massive boost: On Terminal Bench 2 we get 89.9% with Opus 4.6 / Forgecode harness and the adaptive taxonomy used with a Best-of-N Judge, outperforming fixed taxonomies by 15%. (1/n)