Building Evals on Production Data: 🦄 AI That Works #79 x.com/i/broadcasts/1…
Oct 20, 2026 · 5:18 PM UTC
7
8
107
18,152
@dexhorthy Evals on production data are the only only only ones I really trust.
Lab-run leaderboards have become a test where the student also wrote the exam.
Lab-run leaderboards have become a test where the student also wrote the exam.
0
0
1
287
@dexhorthy This is episode #78, right? boundaryml.com/podcast/2026-1… Anyway, banger as always 💪
0
0
0
155
@dexhorthy “用线上数据做评测”这个题目太实用了，离线数据集再全也覆盖不到真实用户的奇怪输入。这期有讲怎么从线上流量里挑样本吗？比如按用户负反馈、低分输出来抽，还是随机抽一批人工标注后再固化成回归集？
0
0
0
89