How We Moved Discord Voice to the Edge

This is a very engaging write-up and I really appreciate companies writing posts that scratch the technical itch. Please keep them coming!

One thing:

Cloudflare identified the issue. To address it, we deployed a CPU affinity change in our container entrypoint by using taskset to pin our worker threads off the vCPU Cloudflare had pinned the Receive IRQ to, and leveraged Receive Packet Steering to spread softirq processing across other cores while keeping workers off them.

This is how I see it looks like:

Hypervisor on host CPU -> guest VM with guest vCPUs -> core-pinned guest containers

But one thing that I don't necessarily read in the article: are the vCPUs themselves pinned by the hypervisor (ex: via qemu-affinity)? I'm going to assume yes, since they write earlier:

Flame graphs taken after our recv, and send fix showed both net_rx_action and local_bh_enable frames lining up exactly with the residual spikes.

But if they guest's vCPUs are not pinned, that's also a source of unreliability. From my experience with qemu + kvm (I'm going to guess Cloudflare have their in-house hypervisor), guest's vCPUs rarely jump around physical cores, but they do jump. It shows up as L1/L2 cache misses (plus htop also shows them).

I read it as "cloudflare pins the receive IRQ to a particular vCPU", instead of the vCPU itself is pinned.