RT @jaseweston: Claim: Humans & AI are better together if we want to recursively (co-)improve to a brighter future for *humanity*
Instead…
Instead…
0
67
0
There's a lot of discussion about the capabilities of AI models. Considerably less attention is paid to the efficacy of the systems we build around them.
Recent work from Hugging Face illustrates just how much this matters: the same model achieved 33% versus 62% task accuracy depending on the agent harness. Optimizing model selection in isolation is insufficient. A cheaper model doesn't necessarily translate into a cheaper or faster system.
At Sigma, we invest heavily in harness engineering and rigorous evaluation, continuously optimizing for accuracy, latency, and cost.
Our latest work on Sigma Assistant is a great example. By rearchitecting our build agent, we improved accuracy from 71% to 86%, completed builds in a third of the time, and reduced inference costs by 3x on complete application builds.
Two things are particularly important to me:
First, these efficiency gains translate directly into savings for our customers, who pay for their own AI inference.
Second, improvements of this magnitude require a mature evaluation discipline that allows us to experiment with different architectures, measure their impact, and confidently ship changes. For this comparison alone, we evaluated both architectures across 130 tasks in 30 evaluation suites, with three trials per task, spanning incremental edits through complete dashboard and application builds.
Better models will continue to raise the ceiling of what's possible. Better harnesses—and the evals that guide their development—determine how much of that potential we deliver to customers.
Great work by the Sigma Assistant team and a fantastic write-up by Halden Lin and Christopher Mouri!
Sigma research: sigmacomputing.com/blog/sigma-ass…
Recent work from Hugging Face illustrates just how much this matters: the same model achieved 33% versus 62% task accuracy depending on the agent harness. Optimizing model selection in isolation is insufficient. A cheaper model doesn't necessarily translate into a cheaper or faster system.
At Sigma, we invest heavily in harness engineering and rigorous evaluation, continuously optimizing for accuracy, latency, and cost.
Our latest work on Sigma Assistant is a great example. By rearchitecting our build agent, we improved accuracy from 71% to 86%, completed builds in a third of the time, and reduced inference costs by 3x on complete application builds.
Two things are particularly important to me:
First, these efficiency gains translate directly into savings for our customers, who pay for their own AI inference.
Second, improvements of this magnitude require a mature evaluation discipline that allows us to experiment with different architectures, measure their impact, and confidently ship changes. For this comparison alone, we evaluated both architectures across 130 tasks in 30 evaluation suites, with three trials per task, spanning incremental edits through complete dashboard and application builds.
Better models will continue to raise the ceiling of what's possible. Better harnesses—and the evals that guide their development—determine how much of that potential we deliver to customers.
Great work by the Sigma Assistant team and a fantastic write-up by Halden Lin and Christopher Mouri!
Sigma research: sigmacomputing.com/blog/sigma-ass…
2
2
4
86
This summer, four PhD researchers joined Sigma AI Labs and built systems that learn from experience. For Sigma customers, that means an assistant that grows more fluent in their business over time and takes on harder reasoning tasks while getting faster and cheaper.
I’m proud of what this group accomplished in a single summer, and grateful to William Sun, Howard Prioleau, Alina Gorbunova, and Leon Tran for an incredible few months 🎉.
Read their recap: sigmacomputing.com/blog/ai-labs-s…
If you’re a PhD student who finds this work exciting, we’re hiring summer 2027 interns in San Francisco and New York.
I’m proud of what this group accomplished in a single summer, and grateful to William Sun, Howard Prioleau, Alina Gorbunova, and Leon Tran for an incredible few months 🎉.
Read their recap: sigmacomputing.com/blog/ai-labs-s…
If you’re a PhD student who finds this work exciting, we’re hiring summer 2027 interns in San Francisco and New York.
Calling all PhD students: Sigma AI Labs is hiring summer 2027 interns in SF and NYC.
This summer’s cohort worked on problems at the center of our research agenda, including agents that learn across tasks, language models that produce calibrated predictions, and deep research heir collective contributions have helped shape the future of AI at Sigma.
Read more on what our 2026 class built, and apply for our 2027 programs: sigmacomputing.com/blog/ai-labs-s…
3
1
2
159
Yesterday, we announced that Sigma Assistant Build Mode is now GA! Everyone can now chat their way to a live, interactive dashboard or app in minutes, securely on top of their warehouse.
And starting today, you can prototype dashboards in Claude Dashboards, then bring them into Sigma to refine, extend, and publish.
The barrier to entry for building enterprise-governed dashboards is going to zero!
sigmacomputing.com/blog/claude-da…
And starting today, you can prototype dashboards in Claude Dashboards, then bring them into Sigma to refine, extend, and publish.
The barrier to entry for building enterprise-governed dashboards is going to zero!
sigmacomputing.com/blog/claude-da…
0
0
3
137
Sigma Agents is now generally available. For me, this launch is also a story about the engineering culture at Sigma.
Sigma Agents started with a quick prototype called “chat element”: a way to chat with the data in your workbook and take actions.
The prototype generated so much excitement internally that we started sharing it with customers early. Their feedback helped shape it through private beta, public beta, and now GA. Since the public beta launch, Sigma Agents was used by 1000+ organizations to build 6,000+ agents that powered 500,000+ conversations.
The product today goes far beyond that first prototype. Many people have put tremendous work into expanding what it can do and making it ready for enterprise use. But the essence of that original idea lives on.
Giving engineers the space to imagine what we should build—and the support to bring it to customers—is one of the things I’m most proud of about Sigma’s culture.
Congratulations to everyone who helped bring Sigma Agents to GA.
sigmacomputing.com/blog/sigma-age…
Sigma Agents started with a quick prototype called “chat element”: a way to chat with the data in your workbook and take actions.
The prototype generated so much excitement internally that we started sharing it with customers early. Their feedback helped shape it through private beta, public beta, and now GA. Since the public beta launch, Sigma Agents was used by 1000+ organizations to build 6,000+ agents that powered 500,000+ conversations.
The product today goes far beyond that first prototype. Many people have put tremendous work into expanding what it can do and making it ready for enterprise use. But the essence of that original idea lives on.
Giving engineers the space to imagine what we should build—and the support to bring it to customers—is one of the things I’m most proud of about Sigma’s culture.
Congratulations to everyone who helped bring Sigma Agents to GA.
sigmacomputing.com/blog/sigma-age…
0
2
4
3,265
Marwan Mattar retweeted
📢 We’re excited to announce the first round of 20+ speakers for TransformX 2022.
Join us in October to hear from the world's leading AI visionaries from companies like @opensea, @OpenAI, @Stellantis and many more!
🎟️Save Your Spot: bit.ly/3B6iR5A
Join us in October to hear from the world's leading AI visionaries from companies like @opensea, @OpenAI, @Stellantis and many more!
🎟️Save Your Spot: bit.ly/3B6iR5A
1
16
68
My first blog post @scale_AI that briefly summarizes and compares a number of fantastic MLOps / Production ML courses that can help folks get started.
I’m also planning on writing a few technical blog posts on production machine learning.
I’m also planning on writing a few technical blog posts on production machine learning.
Our Director of AI/ML, @marwanmattar, shares 4 MLOps Training Courses to help jump-start or supplement your knowledge of production ML. Learn how to choose the right #MLOps course in #AIExchange.👇
bit.ly/3xDmuhM
1
1
3
Great work from @aerinykim and the ML team at @scale_AI!
"GlideNet" accepted at #CVPR2022
I'd like to highlight some interesting aspects of our work.
1. In real-world applications, it's critical to assign visual attributes (color, shape, motion, etc.) to detected objects. This is usually formulated as a multi-label classification
0
0
1
Love @TeamManara's mission! Consider becoming a volunteer or mentor, even if it's just for an hour each month.
Help (women) engineers from #Palestine get into top tech companies like @Google @Meta @amazon. Volunteer as a technical mentor, mock interviewer, or English coach. APPLY BY DEC 1: bit.ly/3qQtILS
0
0
0
Marwan Mattar retweeted
On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning
deepai.org/publication/on…
by Andrew Cohen et al. including @marwanmattar
#ReinforcementLearning #SupervisedLearning
deepai.org/publication/on…
by Andrew Cohen et al. including @marwanmattar
#ReinforcementLearning #SupervisedLearning
0
1
2
Marwan Mattar retweeted
AnimalAI 3-beta is now public. Upgrades to @unity3d ML-Agents v2.0 and supports raycasts, more obs info and all old tasks.
github.com/mdcrosby/anima…
This release provides a ml-agents 2 version that is similar to previous AAI before we add new features that break compatibility.
github.com/mdcrosby/anima…
This release provides a ml-agents 2 version that is similar to previous AAI before we add new features that break compatibility.
0
19
40
Hot off the press! New blog post on a number of fantastic improvements to ML-Agents that we've rolled out over the past 2-3 months. on.unity.com/3o0Ftw5
2
5
31
Marwan Mattar retweeted
New ML-Agents release includes two big new features: self-attention for variable-length observation spaces, *and* a new multi-agent algorithm for cooperative behavior with variable numbers of agents. If you use the toolkit, let us know what you think! github.com/Unity-Technolo…
0
27
142
Marwan Mattar retweeted
Release 14 of ML-Agents is a-live! 🤖
You can now easily create bots for games that use the InputSystem package, no extra input handling necessary. Download and learn more about the release here: on.unity.com/30sOeE7
You can now easily create bots for games that use the InputSystem package, no extra input handling necessary. Download and learn more about the release here: on.unity.com/30sOeE7
2
30
151
Marwan Mattar retweeted
We're back! In our latest video @aktwelve takes the single coin collector @unity #MLAgents project from the last tutorial and shows how you can swap in a camera for visual input to the agent instead of using ray perception.
#Unity3D #ai #machinelearning
youtu.be/7FHyqzUBzZ0
#Unity3D #ai #machinelearning
youtu.be/7FHyqzUBzZ0
0
2
7
Marwan Mattar retweeted
✨ Mobile developer SundayToz freed up one day per week of #gamedev time for their hit game @anipang4, leaving them more time to find the fun!
Discover how they did it 👇 on.unity.com/3putqGd
Discover how they did it 👇 on.unity.com/3putqGd
0
6
30
Marwan Mattar retweeted
This is the COOLEST. A Google Earth-type representation of the planet. Every green dot is a radio station. Click any dot to listen in. It’s like cultural teleportation. You could spend hours with this thing… radio.garden
1,853
30,955
101,810
Looks great @RobosergTV!
I find it amazing what #AI trained in only 4h (50M steps) with #ReinforcementLearning can achieve. After playing #RocketLeague for 3 years I am nowhere close to this skill level.
#madewithunity #unity3d #MLAgents
1
1
3