HTTP desync in Discord's media proxy: Spying on a whole platform
65 points by videah
65 points by videah
no halfway-decent request library would let you inject control characters into your messages.
You'd be surprised! I have seen this exact bug many times.
GCP's classic application load balancer used to be full of these bugs until pretty recently. Same with AWS CloudFront. Might still be :)
I don’t fully understand the failure here. It’s it that the load balancer has some sort of underflow / overflow bug that leaks traffic? I’m thinking of wall bleed and cloud bleed
It's not a memory safety problem; it's incorrect protocol implementation.
AFAICT from reading the article, Discord rolled their own HTTP proxy that takes client requests and coalesces them into a few persistent connections to another HTTP server on GCP.
i.e.,
client <----> discord custom proxy <----> gcp
That proxy forwards requests without sufficient validation, which opens the door to a bunch of fun attacks.
For example, a malicious client can send a partial request like this:
POST /whatever HTTP/1.1\r\n
Host: discord.storage.googleapis.com\r\n
Content-Length: 1000\r\n
\r\n
and instead of waiting for the remaining 1000 bytes, the proxy will forward that request fragment as though it's a complete request. GCP, on the other side of the proxy, will then consider the next 1000 bytes on the connection (i.e., the next few clients' requests) to be part of the malicious client's request body. If the POST request creates a resource accessible to the malicious client, then they can see the contents of the other clients' requests.
I gave a talk on this a while ago that explains it pretty well (I think). See here.
The blog links the PortSwigger write-up on HTTP desync attacks that goes into detail. Their HTTP Request Smuggling document shows the request flow more at the protocol level.
There is an argument that this kind of problem is solved by re-framing the content with a simpler protocol...
But all the "simpler protocols" are either insufficiently complex (SCGI, CGI)[1], or too complex (FastCGI). It seems kind of like a tough problem.
I've heard that HTTP/2 is a good option here because the binary protocol makes it harder to fuck up in this particular way. But it also looks to be at the FastCGI level of complexity.
It would be nice to DIY a protocol but you need support from things like nginx for that to work, so you're stuck with a bunch of bad options.
[1]: Forking a process for every request can be sub-optimal if you're trying to do certain kinds of things quickly. SCGI on the other hand has the problem that the entire request must be buffered before it can be forwarded. Let's say you're writing an authenticated file-upload website, you either have to implement chunking yourself above the HTTP level (at your application level, which is not always ideal), or accept up to the absolute maximum upload size for every request before your application can authenticate to determine if the upload should be permitted. FastCGI (of course depending on how your gateway implements it) lets you "solve" this problem by chunking the request at a lower level, meaning that even if someone does an un-chunked HTTP POST request, you get chunks which include the headers early, allowing you to determine if the request is authenticated early, and therefore allowing you to reject the request early and drop/never store the subsequent chunks anywhere.
It would be nice to DIY a protocol but you need support from things like nginx for that to work, so you're stuck with a bunch of bad options.
But nginx (and Apache) are open source, so you could add support to it. But I suppose that the way things work today, people expect upstream to maintain the code, because it then passes their (or you in this case) operational costs elsewhere.