CALL FOR AI DETECTION BENCHMARKS
I believe it is vital that third parties are able to audit AI detectors like Pangram. Our largest sample size benchmark, from UChicago, was run on Pangram 3 and is a year old now. New LLMs are being released all the time, and the binary task doesn't capture the full spectrum of real world performance. A great benchmark will help shape how the world interacts with AI over the coming years.
I am putting out a call for new third-party evaluations that attempt to answer one or more of the following questions:
- How well do current detectors catch text from the latest frontier AI models? What about in high context settings, like deep research or an agent harness?
- How well do current detectors do on mixed authorship text? Ideally, research in this area explores the full spectrum of AI assistance from light grammatical edits to full co-authorship.
- Any sort of sliced evals around language, background, or ethnicity. Null results that show no bias are still interesting to us.
What I value in a benchmark is extraordinarily clean data, frontier LLM data, and a high enough difficulty that the benchmark is not saturated by most detectors. If you want credits to run your benchmark, please apply here: pangram.com/contact-us/res…
I believe it is vital that third parties are able to audit AI detectors like Pangram. Our largest sample size benchmark, from UChicago, was run on Pangram 3 and is a year old now. New LLMs are being released all the time, and the binary task doesn't capture the full spectrum of real world performance. A great benchmark will help shape how the world interacts with AI over the coming years.
I am putting out a call for new third-party evaluations that attempt to answer one or more of the following questions:
- How well do current detectors catch text from the latest frontier AI models? What about in high context settings, like deep research or an agent harness?
- How well do current detectors do on mixed authorship text? Ideally, research in this area explores the full spectrum of AI assistance from light grammatical edits to full co-authorship.
- Any sort of sliced evals around language, background, or ethnicity. Null results that show no bias are still interesting to us.
What I value in a benchmark is extraordinarily clean data, frontier LLM data, and a high enough difficulty that the benchmark is not saturated by most detectors. If you want credits to run your benchmark, please apply here: pangram.com/contact-us/res…
27
33
370
20,022
@max_spero_ Is there a resource you can point me towards which says what Pangram has been trained on? An eval set/benchmark would require me to find data that Pangram models have not seen during its training.
0
0
0
3
been working out every single day for the past week in an effort to lose some fat and get in better health :D
3
0
8
315
I wonder why Adani chose Kolkata as the place to file the lawsuit? Thoughts welcome.🙏
483
974
7,899
294,294
@rishdotuk @sabeer @MahuaMoitra Do u think Mohua maitra is a huge leader here in WB?
Her actions have zero impact on ground
Her actions have zero impact on ground
1
0
1
48
@AdittyaDeb19058 @sabeer @MahuaMoitra She is still relevant enough for the district administration to stop her from she is relevant nationally. :)
0
0
0
1
Rishu Kumar retweeted
I’m looking for emergency reviewers for 4 ICLR papers on multilingual representation learning, retrieval, reasoning trace monitoring, and LLMs for graphs. Please reach out if you can help!
0
2
1
307
@Rani_Bilizabeth You are the second person I know who had to cancel plans for diff. I guess I gotta look into going to diff next year as well.
0
0
0
1
Rishu Kumar retweeted
A potentially controversial thought: We should make AI PhD coursework only optional.
So many of our PhDs are tied in numerous classes, but:
- Self-learning is easier than ever.
- Many incoming PhDs are already strong researchers.
- We need to free Profs' time for more research mentorship.
- Urgent AI problems need free-thinking minds (like our academic students!), independent of industry for-profit directions.
Courses are valuable. But they don't need to be mandatory, especially when so many problems with AI are threatening our society, and urgently need to be solved.
So many of our PhDs are tied in numerous classes, but:
- Self-learning is easier than ever.
- Many incoming PhDs are already strong researchers.
- We need to free Profs' time for more research mentorship.
- Urgent AI problems need free-thinking minds (like our academic students!), independent of industry for-profit directions.
Courses are valuable. But they don't need to be mandatory, especially when so many problems with AI are threatening our society, and urgently need to be solved.
23
18
228
18,946
Rishu Kumar retweeted
Berlin folks knew what's good in 1885 (at least when it comes to distillation).
3
2
10
392
@adrakchaiiiii I have recently come to know this is a very Indian thing to do. Even on slack/teams or whatever.
0
0
0
37
Our religious festivals have increasingly becoming occasions for alcohol-fuelled celebrations lately.
2
0
4
162
it's crazy how we don't do actual stuff that doesn't involve looking at a screen anymore. after 9 continuous hours of screentime, what do we do to cool down? WE WATCH NETFLIX AND YOUTUBE 😭
HOW HAVE WE NORMALISED THIS
HOW HAVE WE NORMALISED THIS
3
2
48
1,447
@MihikaTries You folks don't just go out and wander or hit the gym? 👀
0
0
0
39
Minister Piyush Goyal may say it jokingly, but there’s a serious point here. In much of Europe, the week’s last meeting is over by Friday 4 pm. When work demands it, Indians will work late into the night and through weekends. It’s a work ethic that gives India an edge.
1,149
266
2,541
372,655
@hvgoenka When work demands it, people in Europe also work nights. Even in Academia.
0
0
0
15
Eggs aren’t even that high in protein compared to other protein sources for people to keep bringing them up when talking about making protein complicated. 100g egg is just 13g protein.
Eggs are cheap, nutritious, and packed with protein. Why do people still make protein so complicated?
98
66
1,718
223,044
@PiyushGoyal Toh ab modi ji foreign trip pe bhi nahi jaayenge? 😍
2
0
16
688
