馃К Bad news for medical LLMs.
This paper finds that top medical AI models often match patterns instead of truly reasoning.
Small wording tweaks cut accuracy by up to 38% on validated questions.
The team took 100 MedQA questions, replaced the correct choice with None of the t the 68 items where a clinician confirmed that switch as correct.
If a model truly reasons, it should still reach the same clinical decision despite that label swap.
They asked each model to explain its steps before answering and compared accuracy on the original versus modified items.
All 6 models dropped on the NOTA set, the biggest hit was 38%, and even the reasoning models slipped.
That pattern points to shortcut learning, the systems latch onto answer templates rather than working through the clinical logic.
Overall, the results show that high benchmark scores can mask a robustness gap, because small format shifts expose shallow pattern use rather than clinical reasoning.
This paper finds that top medical AI models often match patterns instead of truly reasoning.
Small wording tweaks cut accuracy by up to 38% on validated questions.
The team took 100 MedQA questions, replaced the correct choice with None of the t the 68 items where a clinician confirmed that switch as correct.
If a model truly reasons, it should still reach the same clinical decision despite that label swap.
They asked each model to explain its steps before answering and compared accuracy on the original versus modified items.
All 6 models dropped on the NOTA set, the biggest hit was 38%, and even the reasoning models slipped.
That pattern points to shortcut learning, the systems latch onto answer templates rather than working through the clinical logic.
Overall, the results show that high benchmark scores can mask a robustness gap, because small format shifts expose shallow pattern use rather than clinical reasoning.
206
448
2,423
696,273
@rohanpaul_ai This is a common pattern in the world of AI. People mistake clever pattern matching for intelligence or even superintelligence and trust the benchmarks too much. The reasoning capabilities, especially in spatial and temporal (time), are badly lacking for now, but how soon?
Aug 29, 2025 路 6:07 AM UTC
4
1
16
1,276
@KuittinenPetri @rohanpaul_ai I don't see there similar robustness check on humans. There is also no direct comparison with humans. The best accuracy model in original benchmark drops the least in modified version - 8 percentage points. Would humans do better? They didn't check.
1
0
2
56
@KuittinenPetri @rohanpaul_ai There are many areas where AI performs very well, yet it is underused in the real world. I'd say that, if anything, we do not trust the benchmarks enough!
(I am not saying it鈥檚 perfect)
(I am not saying it鈥檚 perfect)
0
0
1
48
@KuittinenPetri @rohanpaul_ai Average human performance drop for NOTA is 19% (in other studies that these researchers conveniently didn't even address). People mistake clever pattern matching for intelligence...
No conclusion can be made from this "study."
No conclusion can be made from this "study."
0
0
0
38
@KuittinenPetri @rohanpaul_ai LLMs won't ever reason. Have to use another approach. Chain of thought etc. can provide some mitigation but no solutions.
0
0
0
8