Mark Yatskar retweeted
We are bringing back the LLMs for Scientific Discovery workshop to @COLM_conf (in SF this year!!), submit your papers by 23 June!
And we are looking for reviewers! If interested, email lm4sci-2026-organizers@googlegroups.com or DM me here on X!
CFP: lm4sci.github.io/docs/2026/call…
Co-organizers: @shannonzshen @StevenyzZhang @HananeNMoussa @AkariAsai @yatskar @hhsun1 @Diyi_Yang
And we are looking for reviewers! If interested, email lm4sci-2026-organizers@googlegroups.com or DM me here on X!
CFP: lm4sci.github.io/docs/2026/call…
Co-organizers: @shannonzshen @StevenyzZhang @HananeNMoussa @AkariAsai @yatskar @hhsun1 @Diyi_Yang
1
24
117
34,185
Mark Yatskar retweeted
1/ Hiring PhD students at CMU SCS (LTI/MLD) for Fall 2026 (Deadline 12/10) 🎓
I work on open, reliable LMs: augmented LMs & agents (RAG, tool use, deep research), safety (hallucinations, copyright), and AI for science, code & multilinguality & open to bold new ideas!
FAQ in 🧵
I work on open, reliable LMs: augmented LMs & agents (RAG, tool use, deep research), safety (hallucinations, copyright), and AI for science, code & multilinguality & open to bold new ideas!
FAQ in 🧵
17
119
642
148,767
Mark Yatskar retweeted
AutoSDT from @osunlp won the Best Paper Award at the #COLM2025 LLM for Scientific Discovery workshop @lm4sci. Many thanks to the organizers and PC for the recognition. AutoSDT automatically collects data-driven scientific coding tasks at scale for training open models. See gratulations to the student co-leads @YifeiLiPKU @HananeNMoussa (first time attending COLM to present the work), our co-authors and expert team @RonZiruChen @ShijieChen98, @BotaoYu24, Mingyi Xue, Qianheng Zhang, Chen Wei, @ZitongLu, @TzuYao3, @yugizhang, Vishal Dey, Benjamin Burns, @gissong, @XuhuiHuangChem, @ningx005, Ali Payani, @n_kahmed!
😍😍😍🎉🎉🎉
😍😍😍🎉🎉🎉
📢 Introducing AutoSDT, a fully automatic pipeline that collects data-driven scientific coding tasks at scale!
We use AutoSDT to collect AutoSDT-5K, enabling open co-scientist models that rival GPT-4o on ScienceAgentBench!
Thread below ⬇️ (1/n)
11
12
111
16,531
Mark Yatskar retweeted
So excited about LM4Sci #COLM2025! I'll be showing recent breakthroughs in first principles based reasoning and real-world applications in materials design, creative tasks, and tech development.
📅 Just 2 days until LM4Sci #COLM2025! 🧠⚙️
✨ The countdown is heating up!
Today's spotlight: Markus J. Buehler (MIT) (@ProfBuehlerMIT)
on Superintelligence for Scientific Discovery 🧵
0
5
14
2,043
Mark Yatskar retweeted
Traveling to #COLM2025 this week, and here's some work from our group and collaborators:
Cognitive biases, hidden knowledge, CoT faithfulness, model editing, and LM4Science
See the thread for details and reach out if you'd like to discuss more!
Cognitive biases, hidden knowledge, CoT faithfulness, model editing, and LM4Science
See the thread for details and reach out if you'd like to discuss more!
1
8
32
2,026
Mark Yatskar retweeted
📅 Just 4 days until LM4Sci #COLM2025! 🤖🤝🔬
🔥 The countdown continues!
Today's spotlight: Diyi Yang (Stanford)
@Diyi_Yang, on a Human-Centered Perspective on Automating Research 🧵
🔥 The countdown continues!
Today's spotlight: Diyi Yang (Stanford)
@Diyi_Yang, on a Human-Centered Perspective on Automating Research 🧵
1
7
12
25,844
Mark Yatskar retweeted
📅Just 5 days until LM4Sci #COLM2025! 🤖🤝🔬
🔥 The countdown continues!
Today's spotlight: David Sontag (MIT CSAIL, Layer Health)
@david_sontag, on Driving medical discovery through scalable chart review from medical records with LLMs 🧵
🔥 The countdown continues!
Today's spotlight: David Sontag (MIT CSAIL, Layer Health)
@david_sontag, on Driving medical discovery through scalable chart review from medical records with LLMs 🧵
1
2
9
1,763
Mark Yatskar retweeted
📆The full agenda for LM4Sci Workshop (10 Oct 2025, Montreal 🇨🇦): keynotes, panels, poster sessions & more!
lm4sci.github.io/docs/schedule
lm4sci.github.io/docs/schedule
0
6
16
19,143
Mark Yatskar retweeted
Next Friday, I'll speak at the LM for Science workshop at #COLM_2025 about using #AlphaEvolve for mathematical discovery.
Montreal Convention Centre, Room 524A
Friday, October 10, 9:30 am
Followed by a panel starting at 10:30
Montreal Convention Centre, Room 524A
Friday, October 10, 9:30 am
Followed by a panel starting at 10:30
📅 Just 1 week until LM4Sci @ COLM 2025! 🤖🤝🔬
🔥 This week we’re spotlighting our incredible invited speakers at the intersection of AI + scientific discovery. 🔥
First up: Abbas Mehrabian (Google DeepMind) @abbas_mehrabian, on AlphaEvolve for mathematical discovery 🧵1/
0
7
19
5,758
Mark Yatskar retweeted
📅Just 6 days until LM4Sci @ COLM 2025! 🤖🤝🔬
🔥This week, we’re highlighting each of our incredible invited speakers at the intersection of AI + scientific discovery🔥
Introducing Daniel S. Weld (AI2 & UW) @dsweld, on Intelligence Augmentation for Scientific Researchers🧵
🔥This week, we’re highlighting each of our incredible invited speakers at the intersection of AI + scientific discovery🔥
Introducing Daniel S. Weld (AI2 & UW) @dsweld, on Intelligence Augmentation for Scientific Researchers🧵
1
6
15
16,795
Mark Yatskar retweeted
📅 Just 1 week until LM4Sci @ COLM 2025! 🤖🤝🔬
🔥 This week we’re spotlighting our incredible invited speakers at the intersection of AI + scientific discovery. 🔥
First up: Abbas Mehrabian (Google DeepMind) @abbas_mehrabian, on AlphaEvolve for mathematical discovery 🧵1/
🔥 This week we’re spotlighting our incredible invited speakers at the intersection of AI + scientific discovery. 🔥
First up: Abbas Mehrabian (Google DeepMind) @abbas_mehrabian, on AlphaEvolve for mathematical discovery 🧵1/
1
6
8
6,718
Mark Yatskar retweeted
❓ Can AI-generated videos always fool people?
👉 Nope -- we catch them in 94% of cases 😅
🎥 With DeeptraceReward (deeptracereward.github.io), we teach MLLMs to detect human-perceived fake traces in AI-generated videos — and release both training data & model!
🔥 Benchmark:
s on 3.3K AI-generated videos + 3.3K real videos
- Pinpoints visually & temporally grounded traces that reveal "fakeness"
🤖 A 7B reward model trained on this data outperforms GPT-5 by +34.7% at detecting, grounding & explaining deepfake traces.
🧠 Key insight: saying “this is fake” is easy; locating where & when is far harder.
📄 Paper: arxiv.org/abs/2509.22646
This work is done with amazing collaborators at Penn, Princeton, and Stanford! @liusiyi64198 @yinuoxu @lupantech @guangqiusehu @tianboyang @tarananatasagar @christophershen @yikaimao @yuanzheliu @keyushshah @chungunlee @YejinChoinka @james_y_zou
@DanRothNLP @ccb 🙌 A huge shoutout to everyone!
#AI #Deepfakes #Multimodal #ComputerVision #MachineLearning #LLM #TrustworthyAI
👉 Nope -- we catch them in 94% of cases 😅
🎥 With DeeptraceReward (deeptracereward.github.io), we teach MLLMs to detect human-perceived fake traces in AI-generated videos — and release both training data & model!
🔥 Benchmark:
s on 3.3K AI-generated videos + 3.3K real videos
- Pinpoints visually & temporally grounded traces that reveal "fakeness"
🤖 A 7B reward model trained on this data outperforms GPT-5 by +34.7% at detecting, grounding & explaining deepfake traces.
🧠 Key insight: saying “this is fake” is easy; locating where & when is far harder.
📄 Paper: arxiv.org/abs/2509.22646
This work is done with amazing collaborators at Penn, Princeton, and Stanford! @liusiyi64198 @yinuoxu @lupantech @guangqiusehu @tianboyang @tarananatasagar @christophershen @yikaimao @yuanzheliu @keyushshah @chungunlee @YejinChoinka @james_y_zou
@DanRothNLP @ccb 🙌 A huge shoutout to everyone!
#AI #Deepfakes #Multimodal #ComputerVision #MachineLearning #LLM #TrustworthyAI
1
15
92
13,191
Mark Yatskar retweeted
I am happy to share that our paper "Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D" has been accepted to #EMNLP2025! 🎉
4
2
9
1,154
Mark Yatskar retweeted
@emnlpmeeting / #EMNLP2025 Accepted Paper: Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D 🔍
📝 Paper: arxiv.org/abs/2506.01275
This work introduces the first benchmark for contrastive reasoning across four modalities simultaneously. When ideo, image, and 3D inputs, models must identify which best answers a natural language query.
Key contributions:
➡️ 174k training + 2.3k manually verified test samples
➡️ Mixture-of-Models Round-Trip-Consistency filtering for quality
➡️ Comprehensive SOTA model evaluation
Reality Check: Even top models achieve only 56% accuracy overall and 42% on four-modality tasks, revealing a fundamental gap in cross-modal understanding that fine-tuning alone can't bridge.
👥 Authors: Artemis Panagopoulou @artemispng, Le Xue @Le_Xue01, Honglu Zhou @zhou_honglu, Silvio Savarese @silviocinguetta, Ran Xu @stanleyran, Caiming Xiong @CaimingXiong, Chris Callison-Burch, Mark Yatskar @yatskar, Juan Carlos Niebles @jcniebles
#FutureOfAI #EnterpriseAI #MultimodalAI #CrossModalReasoning #Benchmarks #FutureOfAI
📝 Paper: arxiv.org/abs/2506.01275
This work introduces the first benchmark for contrastive reasoning across four modalities simultaneously. When ideo, image, and 3D inputs, models must identify which best answers a natural language query.
Key contributions:
➡️ 174k training + 2.3k manually verified test samples
➡️ Mixture-of-Models Round-Trip-Consistency filtering for quality
➡️ Comprehensive SOTA model evaluation
Reality Check: Even top models achieve only 56% accuracy overall and 42% on four-modality tasks, revealing a fundamental gap in cross-modal understanding that fine-tuning alone can't bridge.
👥 Authors: Artemis Panagopoulou @artemispng, Le Xue @Le_Xue01, Honglu Zhou @zhou_honglu, Silvio Savarese @silviocinguetta, Ran Xu @stanleyran, Caiming Xiong @CaimingXiong, Chris Callison-Burch, Mark Yatskar @yatskar, Juan Carlos Niebles @jcniebles
#FutureOfAI #EnterpriseAI #MultimodalAI #CrossModalReasoning #Benchmarks #FutureOfAI
0
4
7
685
Mark Yatskar retweeted
What if survey-derived rubrics 📋 graded ChatGPT instead of vibes?
We benchmark LLMs & deep research systems across 75 research fields 🩺🧬🦾⚗️🏛️🎭💹: Perplexity deep research wins > 82% of head-to-heads vs the next best!
w/ @realliyifei, @cmalaviya11, and @yatskar
We benchmark LLMs & deep research systems across 75 research fields 🩺🧬🦾⚗️🏛️🎭💹: Perplexity deep research wins > 82% of head-to-heads vs the next best!
w/ @realliyifei, @cmalaviya11, and @yatskar
How well can LLMs & deep research systems synthesize long-form answers to *thousands of research queries across diverse domains*?
Excited to announce 🎓📖 ResearchQA: a large-scale benchmark to evaluate long-form scholarly question answering at scale across 75 fields, using querre mined from survey articles 📚!
Website: cylumn.com/ResearchQA
Paper: arxiv.org/abs/2509.00496
Dataset: huggingface.co/datasets/reall…
Code: github.com/realliyifei/Re…
0
11
17
2,363
Mark Yatskar retweeted
How well can LLMs & deep research systems synthesize long-form answers to *thousands of research queries across diverse domains*?
Excited to announce 🎓📖 ResearchQA: a large-scale benchmark to evaluate long-form scholarly question answering at scale across 75 fields, using querre mined from survey articles 📚!
Website: cylumn.com/ResearchQA
Paper: arxiv.org/abs/2509.00496
Dataset: huggingface.co/datasets/reall…
Code: github.com/realliyifei/Re…
Excited to announce 🎓📖 ResearchQA: a large-scale benchmark to evaluate long-form scholarly question answering at scale across 75 fields, using querre mined from survey articles 📚!
Website: cylumn.com/ResearchQA
Paper: arxiv.org/abs/2509.00496
Dataset: huggingface.co/datasets/reall…
Code: github.com/realliyifei/Re…
1
24
62
9,934
Mark Yatskar retweeted
People at #ACL2025, come drop by our poster today & chat with me about how context matters for reliable language model evaluations!
Jul 30, 11:00-12:30 at Hall 4X, board 424.
Jul 30, 11:00-12:30 at Hall 4X, board 424.
Excited to share ✨ Contextualized Evaluations ✨!
Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓
1
7
22
1,397
Mark Yatskar retweeted
In our new paper, “Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries,” we find that adding just a bit of missing context can reorder model leaderboards—and surface hidden biases. 🧵👇
5
28
156
14,925
Mark Yatskar retweeted
Deadline Extended!
Submit to the LM4Sci Workshop @COLM_conf 2025 in Montreal
🧠 Large Language Modeling for Scientific Discovery (LM4Sci)
📅 New Deadline: June 30
📢 Notification: July 24
📍 Workshop: Oct 10, 2025
📝 Non-archival short (2–4p) & full (up to 8p) papers welcome!
Submit to the LM4Sci Workshop @COLM_conf 2025 in Montreal
🧠 Large Language Modeling for Scientific Discovery (LM4Sci)
📅 New Deadline: June 30
📢 Notification: July 24
📍 Workshop: Oct 10, 2025
📝 Non-archival short (2–4p) & full (up to 8p) papers welcome!
1
4
13
4,585