Let's say you commit a change that makes an application 0.05% slower. No big deal, right?

Well, at the scale of Meta, it is a big deal--a small slowdown for a large application can waste thousands of servers. It's such a big deal that Meta needs a way to catch these performance y figure out if an application is even a hundredth of a percent slower.

The big problem with catching tiny performance regressions is that, well, they're tiny. It's easy for them to be lost in the noise. Any simple approach is going to return 99% false positives because most slowdowns are caused by transitory hardware issues, not actual code changes.

Meta's solution to this problem has three high-level steps:

1. Look for performance regressions not in entire applications, but in individual subroutines. Applications have thousands of subroutines, so a small performance regression for an entire app is likely a large one for the affected subroutine. This way, you're looking for ~5% regressions, not ~0.05% regressions, which makes filtering out noise easier.

2. To make tracing at the coroutine level practical, do stack-trace sampling. Regularly collect stack traces of applications and use them to figure out in which subroutines applications spend their time. This is like conventional performance profiling, but done at massive scale.

3. When a regression is detected, do a root cause analysis to figure out if it's caused by transient issues or code changes. If it's caused by code changes, by analyzing the stack traces of the performance regression and comparing them to recent commits, it should be possible to automatically identify which changes caused the regression!

What's the big takeaway? Performance really matters at scale, and small performance issues add up. The paper claims that this system saves millions of servers a year by detecting tiny performance regressions, which is a staggering number. If your code is used by billions of people, keep it optimized!

THANK YOU to our entire community around the world for making #BlackOps6's launch the biggest Call of Duty opening weekend to date.

Black Ops is back!

Border Collie dogs sort black and white ducks
x.com/visualfeastwan…

Replace your GIFs. They hit different #BlackOps6 giphy.com/callofduty/the…

Top 6 things going on right now:

1️⃣ #BlackOps6 is LIVE
2️⃣ #BlackOps6 is LIVE
3️⃣ #BlackOps6 is LIVE
4️⃣ #BlackOps6 is LIVE
5️⃣ #BlackOps6 is LIVE
6️⃣ #BlackOps6 is LIVE

Real-time audio in games these days has an incredible amount of complexity, whether it's about advanced spatial reverb accounting for physics of the materials, or live mixing to prioritize crucial sounds - learn more here about our approach in @CallofDuty
callofduty.com/blog/2024/10/c…

Cheat dev economy is complex and sprawling, and teams like our TeamRicochet are fighting this relentless fight to make the game safer and more enjoyable to play for our players and all.

Here's their Progress Report 👊
callofduty.com/blog/2024/10/c…

This team has always been incredible at accomplishing poignant and beautiful experiences, harnessing real-time tech to the utmost. It was an honor to help them along the way. Well done, Ves and team! <3

The Neural Light Grid: A Scalable Production-Ready Learned Irradiance Volume is an excellent write up on this topic, check it out! research.activision.com/publications/2…

All keynotes and talks at EGSR 2024 are now available on the YouTube channel: youtube.com/@egsr2024

Michael Vance's presentation on the @CallofDuty Open-Sourcing a USD data set from Call of Duty: Warzone Caldera Map) from @siggraph 2024 #rtradvances is now posted - enjoy!

advances.realtimerendering.com/s2024/index.ht…

See our blog for more: blog.activision.com/activision/202…

This year I did a more in-depth talk on trend from gaming, including players' perspectives trends, technology and platforms in the #rtradvances @siggraph 2024 intro - the talk is now online advances.realtimerendering.com/s2024/index.ht… Enjoy!

The talk from Michał Iwanicki from @Activision Central Tech on Neural Light Grid: Modernizing Irradiance Volumes with Machine Learning is online - check out its insights as well as learnings on ML in rendering at runtime #rtradvances @siggraph 2024
tinyurl.com/advances24-nlg

My SIGGRAPH 2024 slides (with speaker notes) are now available in ARM's website.

Moving Mobile Graphics: Transitioning HypeHype to Physically Based Rendering

community.arm.com/arm-community-…

If you attended #rtradvances at @siggraph 2024 in person, would love to know your thoughts - send in: forms.gle/9QkRoeZG9eEXkG…

In particular, next year it'll be 20 years since Advances started. Would love to hear your thoughts on what you'd like to see (if SIGGRAPH invites us!)

#rtradvances from @siggraph 2024: talk by Simon Taylor from @EA @FrostbiteEngine on "Flexible and Extensible Shader Authoring in Frostbite with Serac" slides are shared:
advances.realtimerendering.com/s2024/index.ht…
Enjoy!

If you are watching the presentations in #rtradvances @siggraph 2024 right now, you can discuss the talks and ask questions to the speaker in our Advances Discord server: discord.gg/Ns2K8arfHA

We are so excited to be sharing this milestone with the industry at large hoping to spur innovation and learning from real-world production datasets! Michael Vance just shared more details about this massive dataset at @siggraph #rtradvances

Wow! 🤩 New production USD dataset “Caldera” from @Activision revealed at #rtradvances #SIGGRAPH2024. Four square miles! @mirror2mask

#rtradvances @siggraph part II session began with @Activision ‘s Michael Vance announcing a release of the @CallofDuty Caldera OpenUSD dataset for academic and non-commercial use