Chasing the 0.005% Ghost

Okay, so if I come to you and say, hey, my app's CPU usage just went up by 0.005 percent—

I would literally laugh you out of the on-call rotation. I wouldn't even look at the dashboard.

Right? It's noise. You breathe on a server rack and the CPU fluctuates 0.005 percent. It's nothing.

Exactly. But Meta just published this paper on their FBDetect system, and that exact number, 0.005 percent, is their actual detection threshold for performance regressions in production. Which sounds completely unhinged until you do the math on their scale.

Yeah, because they have millions of servers. The paper mentioned that just in their serverless platform, catching these microscopic regressions saved them something like four thousand servers a year.

Four thousand servers! Because a tiny leak across half a million machines is a literal data center's worth of hardware just spinning its wheels.

But the statistics of this are what got me. How do you even measure that? If you look at total Linux process CPU, a fraction of a percent is buried in standard deviation. It is statistically invisible.

Right, so they don't look at the process. They measure at the subroutine level. So if you have a massive service, and one tiny subroutine eats up maybe zero point one percent of your total CPU—

Okay.

If that specific subroutine regresses by 0.005 percent of the total, it actually means that subroutine just got five percent slower relative to itself. You see?

Ah, so because you zoom in, the variance is localized. That is actually brilliant. You lower the noise floor just by narrowing the scope. But wait, measuring every single subroutine in production? Instrumenting that would ruin the performance you are trying to save.

Yeah, so they use stack trace sampling fleet-wide. No manual instrumentation. They mostly use eBPF in the kernel to sample the stacks. And the part that absolutely blew my mind was how they handle Python.

Oh man, profiling Python in production is famously terrible. Because you either get the Python interpreter stack, or you get the C stack, but never both.

Yes! And Scalene and other profilers try to guess at the C and C++ libraries, but they can't really cross the boundary accurately. So Meta built this thing called PyPerf.

Using eBPF.

Using eBPF. So they drop a probe in the Linux kernel that looks at CPython, and it reconstructs the end-to-end stack trace by merging the native call stack, the Python virtual call stack, and the native C++ libraries the Python code invoked. All in kernel space.

That is wild. What is the overhead on that? It has to be heavy.

Basically zero. For their big services like FrontFaaS, they sample one stack trace per server every 30 minutes. Even their worst-case scenario, sampling once a second for tiny services, it only hit like zero point eight percent overhead.

Okay, so they have this super precise data. But I kept thinking about false positives. Like, if you are looking at subroutines, what happens if a dev just refactors some code? Like they move a chunk of logic from function A to function B.

Function B suddenly looks like it has a massive performance regression!

Exactly. The overall CPU didn't change, but function B's CPU spiked. That would drive developers crazy with fake alerts.

They call that a cost shift. And they built a specific cost-shift detector to filter them out. It looks at the upstream callers or the whole class, what they call a cost domain. If function B went up, but the overall cost domain stayed flat, the system just ignores it. It knows you just moved furniture around.

It's funny you mention filtering, because the craziest stat in the whole PDF for me was the false positive filtering rate. The system flags nearly four million change points in a month, and they filter out 99.7 percent of them.

Yeah, because production is so noisy! You've got load spikes, server failures, canary tests. And they have this incredible way of dealing with transient issues, the stuff that spikes and then goes away. It's called SAX. Symbolic Aggregate approXimation.

Wait, SAX? I don't think I've heard of that.

It is so cool. It basically discretizes a time series into a text string. So it takes the CPU graph and turns it into letters. Like, values in this range are 'a', this range is 'b'. So a graph shape becomes a string like 'abcdcba'.

No way. So they are doing pattern matching on strings instead of raw numbers to see if a spike went back down to normal?

Exactly. Because comparing raw floating point numbers for 'did this recover' is a nightmare. But if the string goes back to 'a', you know the regression went away. It's so much more robust against random outliers.

That is honestly wild. Between that and the SOMDedup thing—

The Self-Organizing Maps for clustering.

Right, where they group all the related regressions together so you don't get twenty tickets for one bad commit. It just seems like the holy grail of monitoring. The paper said a developer on their serverless platform might only get a FBDetect ticket once every four years.

And when they do get it, it's real. They even try to guess the exact root cause commit using text similarity between the stack trace and the commit messages.

Which works like a third of the time, which honestly in a massive microservice architecture is a miracle. It's just a totally different paradigm of monitoring. Catching the ghosts in the machine before they burn down a small town's worth of electricity.

So next time you see a 0.005 percent bump on your dashboard, don't ignore it.

I mean, I still will. I don't work at Meta. My cloud bill is like forty bucks.

Fair enough.