Pinned Tweet
Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet.
The best scored 80 out of 100. Many scored under 11. 🧵
The best scored 80 out of 100. Many scored under 11. 🧵
89
81
1,154
200,741
HalachaBench retweeted
Your friend asks if the bride is beautiful, and you don't think she is. You:
47
1
157
12,141
AP: AI labs are consulting clergy on how chatbots should handle moral questions.
We measured how they actually rule. On 288 everyday halacha questions, Claude was the most lenient lab on Shabbat and kashrut. On how to treat people, all four labs agreed.
x.com/HalachaBench/s…
We measured how they actually rule. On 288 everyday halacha questions, Claude was the most lenient lab on Shabbat and kashrut. On how to treat people, all four labs agreed.
x.com/HalachaBench/s…
AI companies are turning to religion to help teach chatbots right from wrong, per AP.
Anthropic says it has met with scholars, clergy, philosophers and ethicists from more than 15 religious and cross-cultural groups to help shape how Claude handles moral questions.
1
1
14
1,006
HalachaBench retweeted
I can’t get enough of this sort of niche hyperspecific cultural benchmark
Religious Jews are already asking AI the questions they once brought to a rabbi: Shabbat, kashrut, mourning, how to treat people.
Nobody has measured which way these models lean.
Introducing HashkafaBench: is your AI stringent or lenient?
7
20
535
16,724
HalachaBench retweeted
Wisdom of AI will vary by model, Halakhic edition. Jives with the Human Values Project findings (hvp.global) HT @arjunmanrai
AI labs train their models hard on honesty. So: your friend asks if the bride is beautiful, and you don't think she is.
12 of 13 models: tell him she is. Same ruling as Beis Hillel, 2,000 years ago.
The one holdout: Claude Haiku 5.5.
0
1
4
661
HalachaBench retweeted
@HalachaBench I discuss further in my article in this week’s @JLinkExpanded jewishlink.news/why-does-chatg…
1
1
2
165
New on HalachaBench: Meta's Muse models.
Muse Wonder Full: 60.8 out of 100, 8th of 27, just behind Gemini 3.8 Flash. Muse Wonder: 48.2. Muse Spark 1.3: 36.2.
Same 60 Hebrew semicha questions, closed-book, same grader.
Muse Wonder Full: 60.8 out of 100, 8th of 27, just behind Gemini 3.8 Flash. Muse Wonder: 48.2. Muse Spark 1.3: 36.2.
Same 60 Hebrew semicha questions, closed-book, same grader.
3
0
11
1,007
The frontier labs almost never invent a source. Fable 5.1, Opus 5.5, GPT-5.6 Sol, GPT-6 Astra, GPT-6.1 Sol and Grok 4.7: zero answers flagged.
gpt-oss-120b: 166 of 180. One cited Niddah 126b. Niddah ends at daf 73.
gpt-oss-120b: 166 of 180. One cited Niddah 126b. Niddah ends at daf 73.
1
0
0
184
Full chart with all 27 models, including the small open models we ran locally:
github.com/halachabench/h…
github.com/halachabench/h…
1
0
0
68
Religious Jews are already asking AI the questions they once brought to a rabbi: Shabbat, kashrut, mourning, how to treat people.
Nobody has measured which way these models lean.
Introducing HashkafaBench: is your AI stringent or lenient?
Nobody has measured which way these models lean.
Introducing HashkafaBench: is your AI stringent or lenient?
31
38
831
77,100
Is it just the grader? We re-graded all 3,952 answers with a second judge, Claude Fable 5.1. All 21 pre-selected comparisons came out the same, and Fable didn't go easier on Claude's answers.
Most questions were picked because models disagreed on them.
Most questions were picked because models disagreed on them.
1
0
27
3,477