HTTP RateLimit headers

41 points by fanf

freddyb

We all know why this topic is gaining importance. It’s a f…ing disgrace that the response to unlimited scraping is putting the onus on individual servers. Again.

xjix

From a technical perspective, I'm not sure what else is possible with our current stack. Everyone is vulnerable to DDoS under a certain amount of traffic loading.

We need a internet protocol where requests get cheaper the more common they are so the asymmetry is flipped.
- max-headroom
  
  Ah yes, another use case for IPFS.
  - xjix
    
    I was thinking named data networking (named-data.net) tbh
- mariusor
  
  This would make for a good addition to my ratelimit Go module. Currently relying on Retry-After and X-Retry-After.
- spc476
  
  If there are bots out there that ignore robots.txt, this isn't going to help at all, unless it's guaranteed to crash the bot.
  - fanf
    
    I don’t think bots give a monkey’s about HTTP API headers :-) This is about co-operative load control for traffic from legitimate clients.
- schneems
  
  I wrote a novel (AFAIK) cooperation algorithm for GCRA clients. This post walks through the process https://www.schneems.com/2020/07/08/a-fast-car-needs-good-brakes-how-we-added-client-rate-throttling-to-the-platform-api-gem/
- rfmoz
  I see that RateLimit-Policy tries to cover wide scenarios and that results in a complex header. Some examples taken from the draft:
  
  RateLimit-Policy: "burst";q=100;w=60,"daily";q=1000;w=86400 RateLimit-Policy: "default";q=100;w=10 RateLimit-Policy: "peruser";q=65535;qu="content-bytes";w=10;pk=:sdfjLJUOUH==:
  
  A header must to carry metadata about a request or response, like RateLimit, not a complex policy that belongs to a group of requests, in this case with the same partition key. Headers were designed to be frugal.
  
  But with the current scenario, as suggestion, maybe extend (a bit more) Content-Security-Policy with a simple definition combined with an static file in /.well-known/ratelimit.