Genta Winata retweeted
๐ฅณ Happy to share that PingPong has been accepted to EMNLP 2026 - Main Conference!
Special thanks to all of my collaborators: @hanifmz07, @faridlazuarda, Maryam**, @Shmuhammadd, @nedjmaou, @sathready, @ponguru, @KodaliPrashant, Sunayana**, @gentaiscool, @AyuP_AI, @AlhamFikri
Special thanks to all of my collaborators: @hanifmz07, @faridlazuarda, Maryam**, @Shmuhammadd, @nedjmaou, @sathready, @ponguru, @KodaliPrashant, Sunayana**, @gentaiscool, @AyuP_AI, @AlhamFikri
โ ๏ธNew Paper Alertโ ๏ธ
Introducing PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues
Ever seen a 4-player ping-pong game with chaotic turn-taking? Thatโs a perfect analogy for multi-party polyglot in code-switching conversations.
We introduce PingPong, a ulti-party code-switching dialogues covering five language-combination variations, some of which are trilingual and one that involves a different writing script.๐๐
๐Paper: arxiv.org/pdf/2601.17277
๐คDatasets: huggingface.co/code-switching
Huge thanks to all of my collaborators: @hanifmz07, @faridlazuarda, @Shmuhammadd, Maryam**, @nedjmaou, @gentaiscool, @AyuP_AI, @AlhamFikri.
Check this thread out! ๐งต
#PingPongDataset #CodeSwitching #Multilinguality #Polyglot #NLP #LLM #Benchmark
(1/n)
1
7
29
4,228
Genta Winata retweeted
-- Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection (Oral)
@niu_tianyi , @cyjustinchen , @gentaiscool , Shi-Xiong Zhang, Supriyo Chakraborty, Sambit Sahu, @zhan1624 , @EliasEskin
arxiv.org/abs/2601.09692
#ACL2026 Harbor D: Session 17: Oral Session G: Tues. July 7, 11:00-12:30 PT
x.com/EliasEskin/staโฆ
@niu_tianyi , @cyjustinchen , @gentaiscool , Shi-Xiong Zhang, Supriyo Chakraborty, Sambit Sahu, @zhan1624 , @EliasEskin
arxiv.org/abs/2601.09692
#ACL2026 Harbor D: Session 17: Oral Session G: Tues. July 7, 11:00-12:30 PT
x.com/EliasEskin/staโฆ
๐ข Introducing Routing with Generated Data (RGD), a new setting for annotation-free LLM routing. We study how routers can be trained without any ground-truth labels. We also introduce CASCAL, a novel label-free LLM router that identifies niche skills using consensus-voting and .
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
1
2
10
629
Genta Winata retweeted
2/ #ACL2026 Tues. July 7, 11:00-12:30
Harbor D: Session 17: Oral Session G
Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection
@niu_tianyi, @cyjustinchen, @gentaiscool, Shi-Xiong Zhang, Supriyo Chakraborty, Sambit Sahu, @zhan1624, @mohitban47
arxiv.org/abs/2601.09692
x.com/EliasEskin/staโฆ
Harbor D: Session 17: Oral Session G
Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection
@niu_tianyi, @cyjustinchen, @gentaiscool, Shi-Xiong Zhang, Supriyo Chakraborty, Sambit Sahu, @zhan1624, @mohitban47
arxiv.org/abs/2601.09692
x.com/EliasEskin/staโฆ
๐ข Introducing Routing with Generated Data (RGD), a new setting for annotation-free LLM routing. We study how routers can be trained without any ground-truth labels. We also introduce CASCAL, a novel label-free LLM router that identifies niche skills using consensus-voting and .
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
1
2
5
306
Genta Winata retweeted
๐จ Excited to announce that RGD has been accepted to #ACL2026 Main!
Routing with Generated Data (RGD) is a new LLM routing paradigm where routers estimate skills of models using generated data, without ground-truth labels.
We further introduce CASCAL, a new router for RGD that discovers niche skills via consensus voting + hierarchical clustering, with no ground truth needed.
๐งต๐
Routing with Generated Data (RGD) is a new LLM routing paradigm where routers estimate skills of models using generated data, without ground-truth labels.
We further introduce CASCAL, a new router for RGD that discovers niche skills via consensus voting + hierarchical clustering, with no ground truth needed.
๐งต๐
๐ข Introducing Routing with Generated Data (RGD), a new setting for annotation-free LLM routing. We study how routers can be trained without any ground-truth labels. We also introduce CASCAL, a novel label-free LLM router that identifies niche skills using consensus-voting and .
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
0
16
36
4,695
Genta Winata retweeted
VLMs can easily get distracted by unrelated cultural cues. Happy to present our work on this soon at #CVPR2026๐ฅณ
Working on multilingual VLMs? Consider using our benchmark:
๐arxiv.org/pdf/2511.17004
๐คhuggingface.co/datasets/patriโฆ
Amazing work by @patrickamadeus_ and colleagues!
Working on multilingual VLMs? Consider using our benchmark:
๐arxiv.org/pdf/2511.17004
๐คhuggingface.co/datasets/patriโฆ
Amazing work by @patrickamadeus_ and colleagues!
Excited to share that we have committed our paper โVision-Language Models are Confused Touristsโ to #CVPR2026 (Findings)! ๐บ๐ธ๐
Arxiv: arxiv.org/abs/2511.17004
We question whether current SOTA VLMs remain robust in simple cultural grounding QA when distracting contextual objects e, if you eat chicken schnitzel with Mt. Fuji in the background, will the model fail to recognize it as Japanese katsu?
ConfusedTourists introduces:
๐ 5k+ evaluation samples across 3 cultural item categories, comprising 243 unique cultural items from 57 countries and 11 sub-regions ๐
๐ Evaluation of 14 VLMs across 12 data features ๐ค
๐ Findings showing that simple concept mixing can cause up to a -40% drop in perform ๐
Special thanks to my co-authors @IkhlasulHanif0 , @emthehunt, @gentaiscool, @FajriKoto, and my advisor @AlhamFikri for the valuable contributions along the way!
#multimodal #vlm #multicultural #robustness #evaluation #NLProc #ComputerVision
2
18
70
8,024
Happy to have my first Nature paper. Thank you @CAIS for the collaboration nature.com/articles/s4158โฆ
Last week, Humanityโs Last Exam was published in @Nature.
In just over a year, model scores on HLE have risen from under 5% to nearly 40%.
Thank you to @scale_AI and the 1000+ HLE co-authors for helping policymakers and the public track these rapid advances in AI capabilities.
0
0
21
1,509
Genta Winata retweeted
Last week, Humanityโs Last Exam was published in @Nature.
In just over a year, model scores on HLE have risen from under 5% to nearly 40%.
Thank you to @scale_AI and the 1000+ HLE co-authors for helping policymakers and the public track these rapid advances in AI capabilities.
In just over a year, model scores on HLE have risen from under 5% to nearly 40%.
Thank you to @scale_AI and the 1000+ HLE co-authors for helping policymakers and the public track these rapid advances in AI capabilities.
9
41
154
27,903
Genta Winata retweeted
๐ข Introducing Routing with Generated Data (RGD), a new setting for annotation-free LLM routing. We study how routers can be trained without any ground-truth labels. We also introduce CASCAL, a novel label-free LLM router that identifies niche skills using consensus-voting and .
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
1
27
45
11,698
๐กHave you ever wondered whether visionโlanguage models can be easily tricked by adding landmarks or flags to an image?
In the spirit of the holidays๐, we show that VLMs can indeed be easily confused like "Confused Tourists" โ๏ธ: their performance drops significantly when such image perturbations are applied.
๐ Check out "VLMs are Confused Tourists" โ๏ธ here arxiv.org/pdf/2511.17004
#vision #nlproc #robustness
In the spirit of the holidays๐, we show that VLMs can indeed be easily confused like "Confused Tourists" โ๏ธ: their performance drops significantly when such image perturbations are applied.
๐ Check out "VLMs are Confused Tourists" โ๏ธ here arxiv.org/pdf/2511.17004
#vision #nlproc #robustness
Craving holiday-themed paper? Say less๐
Turns out, Vision Language Models are Confused Tourists โ๏ธ๐ตโ๐ซ
We show that adversarially induced cultural scenes significantly impair VLM cultural comprehension and trigger potential bias
#NLProc #multimodal #robustness
/thread ๐งต(1/8)
0
3
16
2,578
Genta Winata retweeted
Craving holiday-themed paper? Say less๐
Turns out, Vision Language Models are Confused Tourists โ๏ธ๐ตโ๐ซ
We show that adversarially induced cultural scenes significantly impair VLM cultural comprehension and trigger potential bias
#NLProc #multimodal #robustness
/thread ๐งต(1/8)
Turns out, Vision Language Models are Confused Tourists โ๏ธ๐ตโ๐ซ
We show that adversarially induced cultural scenes significantly impair VLM cultural comprehension and trigger potential bias
#NLProc #multimodal #robustness
/thread ๐งต(1/8)
3
21
50
20,064
Genta Winata retweeted
One last call for interested applicants! Applications for the SEACrowd Apprentice Program 2026 will end on December 17, 23:59 (UTC-12). See the post below for more details.
๐ Applications are now open for the SEACrowd Apprentice Program 2026!
Join a 3โ4 month guided research journey where youโll collaborate with mentors.
๐๏ธ Apply between Nov 17 โ Dec 17, 2025 (UTC-12)
๐ Program runs Feb โ Jun 2026
seacrowd.org/apprenticeship
0
3
12
1,461
Genta Winata retweeted
Weโve just shared a full retrospective of the first batch our Apprentice Program from research wins to real lessons learned along the way. Itโs inspiring to see how our SEA mentees transformed ideas into papers and datasets through mentorship.๐๐
seacrowd.org/posts/apprentiโฆ
seacrowd.org/posts/apprentiโฆ
1
9
15
950
Genta Winata retweeted
Excited to be at #NeurIPS 2025 in San Diego this week! ๐ด๐ค
Iโm presenting our UniReps workshop paper on Dec 6: Leveraging Parameter Space Symmetries for Reasoning Skill Transfer in LLMs.
Come say hi! Would love to connect and exchange ideas
Paper: arxiv.org/abs/2511.10850
Iโm presenting our UniReps workshop paper on Dec 6: Leveraging Parameter Space Symmetries for Reasoning Skill Transfer in LLMs.
Come say hi! Would love to connect and exchange ideas
Paper: arxiv.org/abs/2511.10850
1
1
1
323
Genta Winata retweeted
๐ Applications are now open for the SEACrowd Apprentice Program 2026!
Join a 3โ4 month guided research journey where youโll collaborate with mentors.
๐๏ธ Apply between Nov 17 โ Dec 17, 2025 (UTC-12)
๐ Program runs Feb โ Jun 2026
seacrowd.org/apprenticeship
Join a 3โ4 month guided research journey where youโll collaborate with mentors.
๐๏ธ Apply between Nov 17 โ Dec 17, 2025 (UTC-12)
๐ Program runs Feb โ Jun 2026
seacrowd.org/apprenticeship
2
32
149
110,006
Genta Winata retweeted
๐ Following R3, we are thrilled to introduce mR3: Multilingual Rubric-Agnostic Reward Reasoning Models ๐
TL;DR: mR3 is a multilingual version of R3, a rubric-agnostic reward reasoning model trained on 72 languages, achieving the broadest language coverage in reward modeling to date. mR3 is able to surpass much larger models, such as GPT-OSS-120B, while being up to 9x smallerโก!
๐ Paper: arxiv.org/abs/2510.01146
๐ป Code: github.com/rubricreward/mโฆ
๐ค mR3 Models: huggingface.co/collections/ruโฆ
๐ค mR3 Datasets: huggingface.co/collections/ruโฆ
Huge thanks to all of my collaborators: @HLeiTR,
@Zilu_Tang_Peter, @ Annie Lee, @DerryTWijaya1, @llamagrp, and @gentaiscool for all of their efforts into creating this work!
Check this thread out! #mR3 #RewardModel #NLP #LLM #Metrics #Alignment #Preference (1/4)
TL;DR: mR3 is a multilingual version of R3, a rubric-agnostic reward reasoning model trained on 72 languages, achieving the broadest language coverage in reward modeling to date. mR3 is able to surpass much larger models, such as GPT-OSS-120B, while being up to 9x smallerโก!
๐ Paper: arxiv.org/abs/2510.01146
๐ป Code: github.com/rubricreward/mโฆ
๐ค mR3 Models: huggingface.co/collections/ruโฆ
๐ค mR3 Datasets: huggingface.co/collections/ruโฆ
Huge thanks to all of my collaborators: @HLeiTR,
@Zilu_Tang_Peter, @ Annie Lee, @DerryTWijaya1, @llamagrp, and @gentaiscool for all of their efforts into creating this work!
Check this thread out! #mR3 #RewardModel #NLP #LLM #Metrics #Alignment #Preference (1/4)
1
4
9
870
Extremely proud to share that our paper: ๐๐: ๐ ๐๐จ๐จ๐ฅ-๐๐ซ๐ข๐๐ง๐ญ๐๐ ๐๐จ๐ง๐ฏ๐๐ซ๐ฌ๐๐ญ๐ข๐จ๐ง๐๐ฅ ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ฎ๐ฅ๐ญ๐ข-๐๐ฎ๐ซ๐ง ๐๐ ๐๐ง๐ญ๐ข๐ ๐๐ฅ๐๐ง๐ง๐ข๐ง๐ has been accepted into @NeurIPSConf 2025!
We introduce T1, a tool-augmented, multi-domain, al dataset specifically designed to capture and manage inter-tool dependencies across diverse domains with a focus on the travel space.
I would like to shoutout all of my amazing co-authors Amartya Chakraborty, Nadia Bathaee, @pareshdashore19, Anmol Jain, @zzeet, Shi-Xiong ZHANG, Sambit Sahu, and Milind Naphade who all worked incredibly hard to make this paper possible!
Paper Link: arxiv.org/pdf/2505.16986
#nlproc #ml #agents
We introduce T1, a tool-augmented, multi-domain, al dataset specifically designed to capture and manage inter-tool dependencies across diverse domains with a focus on the travel space.
I would like to shoutout all of my amazing co-authors Amartya Chakraborty, Nadia Bathaee, @pareshdashore19, Anmol Jain, @zzeet, Shi-Xiong ZHANG, Sambit Sahu, and Milind Naphade who all worked incredibly hard to make this paper possible!
Paper Link: arxiv.org/pdf/2505.16986
#nlproc #ml #agents
2
2
19
900
Key Highlights:
- ๐๐, a comprehensive multi-turn dataset consisting of 13.5k dialogues designed to evaluate tool using, LLM-based agents across nine key domains. The dataset covers a wide range of interaction scenarios, including single-domain, mixed-domain, and fully multi-domain conversations. It incorporates 14 distinct tools, enabling realistic and fine-grained assessment of agent capabilities in complex, tool-driven dialogue tasks.
- Cross-domain tasks and interdependent tool calls within the dataset to increase complexity and realism of the dataset, requiring agents to reason about tool selection and execution order within context.
- ๐๐-๐๐๐๐๐, An LLM based code-generation system, built on open-source language models and equipped with a caching mechanism for improved performance. This architecture enables scalable evaluation and provides a robust, efficient framework for tool-using agents.
- ๐๐, a comprehensive multi-turn dataset consisting of 13.5k dialogues designed to evaluate tool using, LLM-based agents across nine key domains. The dataset covers a wide range of interaction scenarios, including single-domain, mixed-domain, and fully multi-domain conversations. It incorporates 14 distinct tools, enabling realistic and fine-grained assessment of agent capabilities in complex, tool-driven dialogue tasks.
- Cross-domain tasks and interdependent tool calls within the dataset to increase complexity and realism of the dataset, requiring agents to reason about tool selection and execution order within context.
- ๐๐-๐๐๐๐๐, An LLM based code-generation system, built on open-source language models and equipped with a caching mechanism for improved performance. This architecture enables scalable evaluation and provides a robust, efficient framework for tool-using agents.
0
0
1
159