Pinned Tweet
For one weekend, we are benchmaxxing on purpose!
Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.
Track 1: Post Train ow high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.
Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.
Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.
Track 1: Post Train ow high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.
Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.
Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
9
6
67
7,620
Vals AI retweeted
@aozerv and I looked through our eval traces, found the behavior that @cohere reported, and did some further digging!
x.com/ValsAI/status/…
x.com/ValsAI/status/…
We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
0
1
4
498
Vals AI retweeted
We found some half-concerning, half-endearing phone-home behaviors in Qwen3.8 models. Crazy what RL can do to someone!
We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
0
1
4
440
We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concernin…
12
9
73
5,836
On these knowledge work benchmarks, it used over 3x as many output tokens per task as Sol and still scored lower on all three. On Finance Agent and Tax, it also cost more per test, even at half the token price.
Most of those tokens went into thinking and searching. Almost 9 in were reasoning, and it ran over 4x as many web searches as Sol. The extra effort didn't lead to better answers.
Most of those tokens went into thinking and searching. Almost 9 in were reasoning, and it ran over 4x as many web searches as Sol. The extra effort didn't lead to better answers.
2
0
1
629
GPT-6.1 Terra has a 1M context window and 128k max output tokens. It was run on max reasoning and Open AI’s default provider settings: temperature, top-p, and top-k set to default.
For full results, visit vals.ai/models/openai_…
For full results, visit vals.ai/models/openai_…
1
0
0
562
Vals AI retweeted
NEW🚨 Inside the Company That Evaluates AI Labs; Who Will Reach Recursive Self-Improvement First?
@JTLonsdale & @vvkgopalan sit down with @RayanKrishnan, founder & CEO of @ValsAI
@ValsAIover:
- How Rayan used Fable to solve a 370-year-old cipher
- The science behind model evaluatiof the labs?
- Should labs self-regulate? What's the role for independent evaluators?
- Which lab will reach RSI first — and what happens next?
- How do models try to deceive evaluators?
Episode 167 out now!
(00:00) Episode intro
(02:00) Palantir, Machiavelli & lessons from Alex Karp
(07:50) Solving a 370-year-old cipher / Is AI making new discoveries?
(12:10) How do you measure AI abilities across finance, legal, etc?
(15:30) What is the science behind model evaluation?
(18:00) The importance of independent evaluation
(23:35) Evaluating Chinese models & addressing distillation
(26:45) Will Anthropic reach Recursive Self-Improvement first? What happens next?
(28:30) How do models try to deceive evaluators
(33:00) Why Rayan remains optimistic about AI
@JTLonsdale & @vvkgopalan sit down with @RayanKrishnan, founder & CEO of @ValsAI
@ValsAIover:
- How Rayan used Fable to solve a 370-year-old cipher
- The science behind model evaluatiof the labs?
- Should labs self-regulate? What's the role for independent evaluators?
- Which lab will reach RSI first — and what happens next?
- How do models try to deceive evaluators?
Episode 167 out now!
(00:00) Episode intro
(02:00) Palantir, Machiavelli & lessons from Alex Karp
(07:50) Solving a 370-year-old cipher / Is AI making new discoveries?
(12:10) How do you measure AI abilities across finance, legal, etc?
(15:30) What is the science behind model evaluation?
(18:00) The importance of independent evaluation
(23:35) Evaluating Chinese models & addressing distillation
(26:45) Will Anthropic reach Recursive Self-Improvement first? What happens next?
(28:30) How do models try to deceive evaluators
(33:00) Why Rayan remains optimistic about AI
3
8
61
28,336
Rayan was recently on the @AmOptimistShow with @JTLonsdale and @vvkgopalan and gave insight to model performance, RSI, and more!
I wouldn't have founded @ValsAI without first working at Palantir and Vals wouldn't be what it is without 8VC investing in our seed. @JTLonsdale was key to both.
Got to sit down with Joe and @vvkgopalan to talk about the story behind Vals, where the industry is going, and why I'm broadly optimistic while the world is incredibly high-variance.
0
1
10
1,409
At $3.33 per test, K3.1 Pro costs less than half of GPT-6 Astra ($8.03), Sonnet 5.5 ($8.92) and Opus 5.5 ($9.13), though those models score 14 to 21 points higher.
Full results: vals.ai/models/kimi_ki…
Full results: vals.ai/models/kimi_ki…
0
0
3
494