Daniel Chang retweeted
There’s still lots of unsaturated domains left! Proud of Third Moment (@thirdmomentres) being the exclusive provider of quantitative trading environments for the GDPx benchmark.
Nicely done @SemiAnalysis_ @dylan522p
Nicely done @SemiAnalysis_ @dylan522p
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
3
5
16
1,472
Daniel Chang retweeted
It’s time to saturate the unsaturated domains! Was great to contribute to this benchmark on the quantitative trading sector.
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
0
1
4
632
Daniel Chang retweeted
Awesome to have contributed as the cyber partner for @SemiAnalysis_ GDPx benchmark! An incredibly important project to capture the current state of frontier AI's impact on our economy
Hackers are improving quickly, and cyberattacks are becoming more common. Now more than ever, we need to scale defensive capabilities!
@arithmeticlabs1
Hackers are improving quickly, and cyberattacks are becoming more common. Now more than ever, we need to scale defensive capabilities!
@arithmeticlabs1
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
6
3
34
2,426
Daniel Chang retweeted
another day, another llm-made cybersec horror beyond your comprehension
49
408
3,527
856,761
Daniel Chang retweeted
Physical RSI is the next frontier.
Proud of our team at @btfl_machines for bringing real-world hardware design to GDPx.
Proud of our team at @btfl_machines for bringing real-world hardware design to GDPx.
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
6
1
17
199,760
Daniel Chang retweeted
Official apology to Google from me, they cooked with Carbon.
It's spikier in terms of domains and capabilities, but it's really good in a few areas.
It's spikier in terms of domains and capabilities, but it's really good in a few areas.
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
121
138
2,043
245,563
Daniel Chang retweeted
Super excited to finally launch GDPx! @AndrewMegalaa
is a beast who switched to polyphasic sleep last week to get this out in time.
We truly believe this is the best benchmark in the world for measuring frontier model capabilities. And since the best public model today sits at just 12%, we can't wait to see how some rumored upcoming releases perform as well 👀
is a beast who switched to polyphasic sleep last week to get this out in time.
We truly believe this is the best benchmark in the world for measuring frontier model capabilities. And since the best public model today sits at just 12%, we can't wait to see how some rumored upcoming releases perform as well 👀
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
6
9
53
4,833
Daniel Chang retweeted
Tons of fun working with @maxkan and @AndrewMegalaa on this over the past month!
check out the finance tasks @ gdpx.semianalysis.com/domains/finance
check out the finance tasks @ gdpx.semianalysis.com/domains/finance
GDPx covers the following domains. Each was a collaboration with at least one specialized data partner. SemiAnalysis was responsible for final QA and heavily involved with task iteration. (2/7)
4
12
27
996
Daniel Chang retweeted
Proud for @emulated_ai to be a science partner contributing to the frontier of materials science!
With most of my career being in software, part of me has always wanted to go back to physical sciences. Glad to be working with the smartest folks in materials engineering & research!
To push the frontier we need realistic evals fully simulating laboratories and workflows. It takes the best to pull that off
With most of my career being in software, part of me has always wanted to go back to physical sciences. Glad to be working with the smartest folks in materials engineering & research!
To push the frontier we need realistic evals fully simulating laboratories and workflows. It takes the best to pull that off
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
3
8
48
3,949
Daniel Chang retweeted
Happy to announce that GDPx is released!
It was super fun working on the initial benchmark design, rigorous task QC, and creating eval best-practices for the future.
Here's to an earnest, legible, and superhuman future!
It was super fun working on the initial benchmark design, rigorous task QC, and creating eval best-practices for the future.
Here's to an earnest, legible, and superhuman future!
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
12
5
87
6,383
Daniel Chang retweeted
My experience partying all night at Berghain in Berlin made me think of how we're lacking this experience in the San Francisco tech scene. We can literally become the coolest, most tasteful social scene on Earth if we had our version of Berghain, but better. So let's build it. The best aspect of Berhain is how they select who gets in. The club has a vibe and atmosphere, and if you aren't wearing the right clothes or have the right demeanor, you don't get in. That's how you cultivate a tasteful crowd. We can improve on this by only letting in high agency, talented operators in the tech scene who can still pull off dope outfits, so no geeks. SF is in dire need of late night venues, and Berghain goes until 8am. That's a little too crazy for high net worth operators who are changing the world so we could do something like 2am. Berghain also is kinda dark and in a weird building so we'd put the club in a SoMa corporate office that we convert to speakeasy or barcade vibe. Maybe add a quiet area for people to bring their laptops and work? I honestly feel like we're improving on the Berghain form here, especially by swapping out the crowd of weird partiers in inappropriate clothes (or lack there of) with insanely cracked founders and engineers. I'm super stoked about this idea, SF tech could really run the dopest parties on the planet.
292
97
2,546
361,850
Daniel Chang retweeted
I hope (and it seems) that mech interp gets bitter-lessoned, which would be a huge deal. A lot of time was been spent on SAEs (and others ideas) over the last few years, and it seemed like the field wasn't moving fast enough overall (unlike capabilities).
Mech interp continues to be the field with the most interesting research.
Mech interp continues to be the field with the most interesting research.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com
2
3
32
2,084
Daniel Chang retweeted
Today we are finally coming out of stealth with a few announcements:
- Founding @thirdmomentres with my friend Jeffrey Yan: automatically improving inference endpoints (with focus on data research).
- Fundraise at $250M valuation earlier this Summer with investors including @neo , @adamdangelo and more.
- 0 -> 8 figures in revenue within ~6 months of founding the company. In just this past month, we won $10M+ in contracts.
- Hiring ~15 more engineers in the next 45 days. Currently based in SF, opening a NYC office soon.
Jeffrey and I are building Third Moment to do two things: (1) make it effortless for enterprises to deploy state-of-the-art AI tailored to their workflows, and (2) build best-in-class RL environments that push the frontier of model capabilities. The design of a reinforcement learning environment has an underappreciated impact on the resulting behaviors of the models trained on it.
If you are interested in pushing the frontier of ML systems and data research, reach out to us (team@thirdmomentresearch.com). Check out OpenAI’s blog post on some of our experiments in the comments.
- Founding @thirdmomentres with my friend Jeffrey Yan: automatically improving inference endpoints (with focus on data research).
- Fundraise at $250M valuation earlier this Summer with investors including @neo , @adamdangelo and more.
- 0 -> 8 figures in revenue within ~6 months of founding the company. In just this past month, we won $10M+ in contracts.
- Hiring ~15 more engineers in the next 45 days. Currently based in SF, opening a NYC office soon.
Jeffrey and I are building Third Moment to do two things: (1) make it effortless for enterprises to deploy state-of-the-art AI tailored to their workflows, and (2) build best-in-class RL environments that push the frontier of model capabilities. The design of a reinforcement learning environment has an underappreciated impact on the resulting behaviors of the models trained on it.
If you are interested in pushing the frontier of ML systems and data research, reach out to us (team@thirdmomentresearch.com). Check out OpenAI’s blog post on some of our experiments in the comments.
37
20
147
22,092
Daniel Chang retweeted
Proprietary data which is why big 4 accounting firms beat ramp - oh wait
Network effects which is why Github won AI software dev - oh wait
Physical component which is why there are no new defense contractors - oh wait
There is no moat. You are defenseless and also can win it at all at the same time. Act accordingly.
Network effects which is why Github won AI software dev - oh wait
Physical component which is why there are no new defense contractors - oh wait
There is no moat. You are defenseless and also can win it at all at the same time. Act accordingly.
Technical moats are mostly over at this point. If you’re building something that is meant to be durable, the only moats now are:
• Proprietary data
• Network effects
• A physical component
50
68
1,178
69,844
Daniel Chang retweeted
you were and you weren't. exponential progress is actually a series of sigmoid nested together.
3
6
222
6,148
Daniel Chang retweeted
actually insane win for openai. henry is a generational talent
An announcement: I've joined @deanwball's Strategic Futures team at OpenAI to help shape frontier AI policy. Our work will focus on on how transformative AI will reshape the economy, our institutions, and free society, and how human agency can be preserved in the process.
9
8
627
92,582
Daniel Chang retweeted
Today we are announcing Corma, and our $60M seed round led by @sequoia.
@CormaAI is building foundation models toward superintelligence for defensive cybersecurity. We do that because general AI is becoming exponentially more capable in offensive security, while it hasn't been same rate in cyber defense - creating an inherent imbalance between attackers and defenders that otherwise cannot be addressed.
Recent events have shown that building frontier AI models for defensive cybersecurity is one of the crucial unsolved problems of the AI age - and solving it is the generational mission Corma was founded on.
Our foundation model has already far surpassed every general-purpose frontier model on defensive cybersecurity tasks, and we are committed to remaining the frontier of defensive cybersecurity intelligence.
Corma is already protecting Fortune 100 companies and some of the world's most important organizations, across healthcare, finance, critical infrastructure, retail, and more. This is just the beginning, as we see it as our calling to protect every organization in the world that needs frontier defense.
We are grateful to our partners @shaunmmaguire and @sonyatweetybird at @sequoia , @vkhosla at @khoslaventures , @DanRose999 at @coatuemgmt , and @Weiner_Lia at Netz Capital, who believed in our ability to lead the intelligence race in cybersecurity and make sure the defenders always win.
@CormaAI is building foundation models toward superintelligence for defensive cybersecurity. We do that because general AI is becoming exponentially more capable in offensive security, while it hasn't been same rate in cyber defense - creating an inherent imbalance between attackers and defenders that otherwise cannot be addressed.
Recent events have shown that building frontier AI models for defensive cybersecurity is one of the crucial unsolved problems of the AI age - and solving it is the generational mission Corma was founded on.
Our foundation model has already far surpassed every general-purpose frontier model on defensive cybersecurity tasks, and we are committed to remaining the frontier of defensive cybersecurity intelligence.
Corma is already protecting Fortune 100 companies and some of the world's most important organizations, across healthcare, finance, critical infrastructure, retail, and more. This is just the beginning, as we see it as our calling to protect every organization in the world that needs frontier defense.
We are grateful to our partners @shaunmmaguire and @sonyatweetybird at @sequoia , @vkhosla at @khoslaventures , @DanRose999 at @coatuemgmt , and @Weiner_Lia at Netz Capital, who believed in our ability to lead the intelligence race in cybersecurity and make sure the defenders always win.
96
72
714
304,831