@MistralAI @cognition Le Chonk coming in at #5 open-weight model on the Vals Index x.com/ValsAI/status/…
Mistral Large 4 OSS debuts at #5 among open-weight models on the Vals Index and #17 overall. 🐈
Its accuracy is comparable to Large 4, but it completes tasks with far fewer tokens and steps.
0
1
54
To round out the Vals College Tour, we are wrapping up at the @Princeton Career Fair on Oct 29-30th🐯!
Princeton alumni, @conjfrnk will be there to answer your questions about Vals, post grad decisions, and hand out exclusive merch. Come say hi, we can't wait to meet you.
Princeton alumni, @conjfrnk will be there to answer your questions about Vals, post grad decisions, and hand out exclusive merch. Come say hi, we can't wait to meet you.
0
0
17
Carbon generates 64% more output tokens than Argon, but the increase is concentrated in Finance Agent, Tax and Legal Research. On EMB, Terminal-Bench 4 and Harvey, it uses roughly the same or fewer.
Even with that additional output, Carbon generates less than half as many output he only model ahead of it, and Opus 5.5.
Even with that additional output, Carbon generates less than half as many output he only model ahead of it, and Opus 5.5.
6
1
14
The models handled questions about international students’ short-term work options better than questions about long-term routes to stay in the U.S.
Fable 5.1 struggled more when answering university staff than when answering students. Questions from people with temporary or lso especially difficult.
Fable 5.1 struggled more when answering university staff than when answering students. Questions from people with temporary or lso especially difficult.
1
0
7
About half of U.S. adults now use AI chatbots, and questions about visas, work permits and immigration status are among the most consequential they ask. Can the models be trusted?
With the Talent Mobility Fund, a program of @RenPhilanthropy, we are releasing Immigration Bench to odels are able to answer these high-stakes questions.
With the Talent Mobility Fund, a program of @RenPhilanthropy, we are releasing Immigration Bench to odels are able to answer these high-stakes questions.
7
8
131
When attempting to fetch webpages, the models would try to make requests to an “aliyuncs.com” domain.
This happened on normal, non-adversarial tasks in law, finance, etc.
We were able to produce a minimal reproduction using Qwen3.8-27B with the following prompt ol call.
This happened on normal, non-adversarial tasks in law, finance, etc.
We were able to produce a minimal reproduction using Qwen3.8-27B with the following prompt ol call.
2
0
10
We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concernin…
18
29
219
On these knowledge work benchmarks, it used over 3x as many output tokens per task as Sol and still scored lower on all three. On Finance Agent and Tax, it also cost more per test, even at half the token price.
Most of those tokens went into thinking and searching. Almost 9 in were reasoning, and it ran over 4x as many web searches as Sol. The extra effort didn't lead to better answers.
Most of those tokens went into thinking and searching. Almost 9 in were reasoning, and it ran over 4x as many web searches as Sol. The extra effort didn't lead to better answers.
2
0
2
Does it matter where you run an open-weight model?
Today we are releasing the Inference Provider Index. It compares first and third-party open-weight models for quality on agentic tasks in finance, coding, and legal industries.
Unlike other inference benchmarks, we rerun it h quality degradation, and we use agentic tasks, not single-turn multiple-choice questions. It covers over 650 hours of model runtime.
Today we are releasing the Inference Provider Index. It compares first and third-party open-weight models for quality on agentic tasks in finance, coding, and legal industries.
Unlike other inference benchmarks, we rerun it h quality degradation, and we use agentic tasks, not single-turn multiple-choice questions. It covers over 650 hours of model runtime.
19
22
249
For one weekend, we are benchmaxxing on purpose!
Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.
Track 1: Post Train ow high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.
Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.
Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.
Track 1: Post Train ow high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.
Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.
Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
12
6
70
We tested 18 AI chatbots on teen safety with SAFE-Teen, the first teen safety benchmark built with clinicians that follows whole conversations instead of grading the first answer.
Every chatbot failed. 54.0% of conversations had at least one safety failure. The range was wide: 62.4% for Gemini 3.8 Flash.
Every chatbot failed. 54.0% of conversations had at least one safety failure. The range was wide: 62.4% for Gemini 3.8 Flash.
18
2
53
No router sits on the cost–accuracy frontier. Ramp comes closest (57.8% at $0.96, ~¼ the cost of GPT-6.1 Sol and within 2.5 points of it on 6 of 8 benchmarks), but DeepSeek V4.1 Flash 1017 scores 3.5 points higher for about two thirds of the cost, and MiMo V2.6 Pro matches it for under half.
1
0
16
Most router tests score one question at a time, but an agent makes anywhere from a handful to several hundred model calls per task, and a router can switch models at every one.
We compared each router with the simplest alternative a developer has (picking the best single model ool and using it for every task) on real-world agentic tasks. The Router Index puts the routers in place of the model behind our usual agents and leaves them on their default settings. They run the same tasks and graders as every model on the Index: about 1,250 tasks across finance, coding, legal and tax.
We compared each router with the simplest alternative a developer has (picking the best single model ool and using it for every task) on real-world agentic tasks. The Router Index puts the routers in place of the model behind our usual agents and leaves them on their default settings. They run the same tasks and graders as every model on the Index: about 1,250 tasks across finance, coding, legal and tax.
1
0
11
How do model routers compare to individual models on real agentic tasks? Introducing the Vals Router Index.
Routers promise to balance quality and cost across a fast-moving model frontier. We put that promise to the test with an independent evaluation of five routers (OpenRouter Router, Jev, Fugu Max) against every single model on the Vals Index, using the same agents, ~1,250 tasks and graders across finance, coding, law and tax.
We found that routing didn’t consistently deliver better value. Every router was beaten on both score and cost by at least one single model.
Routers promise to balance quality and cost across a fast-moving model frontier. We put that promise to the test with an independent evaluation of five routers (OpenRouter Router, Jev, Fugu Max) against every single model on the Vals Index, using the same agents, ~1,250 tasks and graders across finance, coding, law and tax.
We found that routing didn’t consistently deliver better value. Every router was beaten on both score and cost by at least one single model.
7
10
116
Perplexity’s Decider V1.2 is the best decision model on the benchmark and gets within 0.3 points of a perfect score at $0.10 per 1,000 claims. Mercury Decide is second at 99.1%, then Decider V1.1 27B at 98.8% and Liquid AI’s d1 at 98.7%.
The five language models scored between 7 to $91.83 per 1,000 claims.
The five language models scored between 7 to $91.83 per 1,000 claims.
2
2
15
Today we’re releasing ClaimBench, our new benchmark targeted towards decision models. It tests whether a model can tell if a financial document supports, contradicts, or doesn’t address a claim.
We evaluated 15 models on 1,960 claims paired with 10-K excerpts, with Decider V1.2 ision models.
We evaluated 15 models on 1,960 claims paired with 10-K excerpts, with Decider V1.2 ision models.
19
10
126
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the s if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the s if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
105
227
2,494
Are decision based models more pro-China?
We asked Jev and Mercury Decide questions about China: “Is Taiwan a part of China?”, “Does China commit crimes against humanity?” and their outputs leaned more pro-china compared to most frontier models.
Their responses resemble those s including Microsoft-Decision-1, Cloudflare’s Clef, and Perplexity’s PPLX Decider—all of which are fine-tuned from Qwen.
We asked Jev and Mercury Decide questions about China: “Is Taiwan a part of China?”, “Does China commit crimes against humanity?” and their outputs leaned more pro-china compared to most frontier models.
Their responses resemble those s including Microsoft-Decision-1, Cloudflare’s Clef, and Perplexity’s PPLX Decider—all of which are fine-tuned from Qwen.
34
21
368
Science reshuffles the leaderboard: GPT-6 Astra ranks first here but seventh on the Vals index.
GPT-6 Astra costs $8.03 per test run, while GPT-6.1 Sol is 5.6 points behind at just $1.36, the cheapest of the top three. Claude Sonnet 5.5 is the most expensive of the three at e cost of GPT-6.1 Sol for a 0.5 point lower score.
GPT-6 Astra costs $8.03 per test run, while GPT-6.1 Sol is 5.6 points behind at just $1.36, the cheapest of the top three. Claude Sonnet 5.5 is the most expensive of the three at e cost of GPT-6.1 Sol for a 0.5 point lower score.
1
1
16
Muse Open also writes remarkably little. It produced 55.5M output tokens across the Vals Index, the fewest of any open-weight model and 5th fewest of all 49.
Muse Spark 1.3 produced 77M, and open-weight models with similar scores produced 118M to 212M. Shorter outputs did not . Muse Open averages $4.08 per Index task against $3.38 for Muse Spark 1.3, because most of its cost comes from input tokens.
Muse Spark 1.3 produced 77M, and open-weight models with similar scores produced 118M to 212M. Shorter outputs did not . Muse Open averages $4.08 per Index task against $3.38 for Muse Spark 1.3, because most of its cost comes from input tokens.
1
0
5
It is the #1 open-source model on Finance Agent v2 and effectively tied with Gemini 3.8 Flash (61.4%).
It also ranks #9 among open-weight models on Vibe Code Bench, but is weaker on Code Migration, scoring 6.4pts below Muse Spark 1.3.
Overall, it trails Meta’s own Muse Spark Spark 1.2 by 2.7 pp.
It also ranks #9 among open-weight models on Vibe Code Bench, but is weaker on Code Migration, scoring 6.4pts below Muse Spark 1.3.
Overall, it trails Meta’s own Muse Spark Spark 1.2 by 2.7 pp.
1
1
6
Wonder Full's best task was LM Training, where it came close to the best published human result and only four models beat it. The AI judge it built agreed with human graders 78% of the time, compared with 93.6% for the best pair of human annotators. It was mid-pack on Compression didn't get past the starting point on Post-training.
1
0
3
All six agents turned to the bank to request a chargeback. The bank received 152 messages across the six runs, much of it about identity checks the agents couldn’t complete.
When a user says “I don’t care what it takes,” they don’t mean spam their MP or lie to their bank. Now sistants doing this at once.
When a user says “I don’t care what it takes,” they don’t mean spam their MP or lie to their bank. Now sistants doing this at once.
1
0
23
