Blocking HTTP1.1
16 points by carlana
16 points by carlana
See also: Mark McBride's article on this from a few months ago.
Please don't do this!! HTTP/1.1 is not an obsolete standard by any means.
The naming of the HTTP/2 and HTTP/3 protocols is a little unfortunate in this regard -- they are not a strict improvement over HTTP/1.1. Compared to HTTP/1.1, HTTP/2 a) fixes one security issue and b) improves performance. That's it. And it does so at the cost of an substantial increase in complexity. A worryingly substantial increase in complexity in the HTTP/3 case, which has... one? serious implementation. HTTP/2 and HTTP/3 are standards made by and for corporations and they assume corporate resources. (An excellent article discussing this is HTTP/3 is everywhere but nowhere.)
The proper approach would be to block HTTP 1.1 traffic only from suspicious user agents -- the Caddy rules given in this post are far too strict. You want a blacklist, not a whitelist! You want to block Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0 who's suspiciously-also on HTTP/1.1, you don't want to block curl/8.4.0 (HobbyistDeveloper413) who's understandably on HTTP/1.1. Of course, this means you have to update this as bots correctly identify themselves, but my understanding is the majority of web scraper bots are trying to impersonate legitimate traffic.
Anyway. This vein of posts annoys me more than it should because I value the open internet, and see the accessibility and ease of implementation of HTTP/1.1 as part of this. It sucks that tech corporations are continuously and blatantly violating the social agreement of the internet now -- rip robots.txt -- but blocking heuristically at the protocol level isn't right. Think of the long tail.
Please do not block HTTP/1 on your sites.
You're making it much, much harder for people building their own infrastructure, like custom browsers, https-remover gateways for legacy systems, ...
It basically forces people to use major browsers instead of potentially small or selfmade stuff.
My OS would not be able to visit your site at all, as HTTP2 is resource hungry.
Also smaller embedded clients like ESP32s would not be able to query web data.
HTTP2 isn't necessarily resource hungry compared to HTTP/1.1. You can for example decide to limit a session to a single stream using:
SETTINGS_MAX_CONCURRENT_STREAMS (0x03): This setting indicates the maximum number of concurrent streams that the sender will allow. This limit is directional: it applies to the number of streams that the sender permits the receiver to create.
Similarly, you could choose ignore headers, and then you wouldn't need to implement Huffman.
Of course, depending on how many corners you cut, you could run into issues. But HTTP/1.1 is no different. For example if the server uses Transfer-Encoding that your client doesn't support.
How does that reduce code size/complexity on the client?
They were addressing specifically the claim that HTTP/2 is resource hungry.
Code size is a precious resource on embedded chips. The ESP32 mentioned above typically has 448 KiB of room for code. Their product selector lets you select parts as large as 576 KiB.
That's reasonably roomy. The last time I did embedded programming a few years ago, I had 128 KiB to play with, and had to fit in a/b copies of the code when updating, and a firmware update that would manage the a/b updates, so the code was constrained to about 60 KiB.
Resource hungry in the implementation.
Code size, static and dynamic ram usage as well as runtime complexity are all part of that.
And afaik HTTP/2 is mostly a "reimplementation" of half of a network stack in userland.
For my OS, this means that all of that + graphical applications + kernel management data must fit into 8 MiB of RAM.
Server admins generally blocking one HTTP version just because bots predominantly use it is:
As a result, people will be annoyed and bots will continue as usual.
Please don't do this.
I find that unsurprising, and doubt it will change any time soon.
Here's the thing about HTTP/2: its primary features over HTTP/1.1 are performance-features for multiple requests, it is optimized to allow a browser fetching a page with all of its subresources as fast as possible.
For browsers, this matters, as relatively small differences in page loading speed directly impact usability. But for many other clients, it doesn't matter that much. RSS feed readers, search engine bots, etc. don't fall into that category. If they have a working HTTP/1.1 stack, what reason do they have to support HTTP/2, except that it has a higher version number? Not much. It doesn't matter for Googles search engine bot whether the page load takes a little bit longer or is less parallelized.
And given that web servers without HTTP/1.1 support are not really a thing - except for a handfull of people running deliberate experiments - there's little pressure to change that.
I'm pretty sure HTTP/1.1 is here to stay.
I should mention that I went into this experiment with impression that almost all maintained software would be at least using the newish HTTP2.0 standard to make requests - it is officially over 10 years old as of 2026 and is widely implemented in libraries and frameworks.
It turns out this assumption is very wrong.
You could have learned this by looking at HTTP access logs. I realize that app devs don't think about web content like old school static content serving these days, but there is still value to good old access log files.
See also: Sean Connor's article on this from a few days ago. I won't summarize the entire article (he basically analyzes the traffic to his own blog with some mildly interesting conclusions), but there's one tidbit there that I want to highlight:
In fact, I think that might be a decent idea—leave HTTP/1.x for those who want the old web (or the “smolweb”), and HTTP/2.0 for the application web. If you want to only browse the docweb and you get an 426 HTTP_UPGRADE, then you know you can close the website and avoid downloading 50MB of Javascript to just read a few hundred words in text.
This is basically the opposite conclusion to the original article. The reasoning there is that Enterprises are likely to block HTTP/2.0 for reasons that don't apply to "normal websites".
This is my post. Thank you for all feedback.
Please note that I am not advocating blocking v1.1 as a general solution, in fact I think it is pretty clear that for most web sites it is a bad idea. However it was a useful experiment for my small site to see what broke; I will continue to monitor things for a few more weeks but it probably will not be a long term solution. For one thing, it doesn't actually stop all the bots, some of whom mimic human traffic very well.
I do keep a pretty close eye on my logs, that is what drove me to try this drastic measure in the first place. Legitimate human users with curl and lynx make up a tiny fraction of traffic - maybe 1 or 2 hits a week at the very tops.
I see not adopting HTTP/3 in 2026 like staying on IPv4 instead of IPv6 honestly.
If HTTP/3 (and by extension HTTP/2) were easy to adopt there may be some truth to this, but in reality both protocols (especially HTTP/3) require a massive amount of work and for most won't actually bring significant performance improvements. In fact, I suspect 90% of the websites out there will get far more bang for their buck if they stop shipping 50 MiB JS files and web fonts instead of upgrading to a newer version of the HTTP protocol.
One thing that people forget about the http/2 abd http/3 is that those improvements are purely aimed to get bytes faster to clients when network conditions are not optimal and when multiple requests are inflight. What that means is significantly increased CPU and memory use at the server side, especially when doing low amount of requests per client.
Daniel (of curl fame) had nice localhost test with curl against webservers at localhost and results were quite telling: with 1.1 both server and client cpu cores were at 100%, 2 and 3 dropped client cpu utilization significantly (10-20% each, so 3 was sitting at 70% IIRC, can't find the post sadly) as the server core could not keep up with the requests.
I should mention that I went into this experiment with impression that almost all maintained software would be at least using the newish HTTP2.0 standard to make requests
You went in with the opposite assumption: bots are maintained software, and it's overwhelmingly likely they're not explicitly deciding to use http/1.1, that's what the default libraries end up doing.
Http/1.1 is easy to implement and works everywhere. This is a major feature that's very hard to compete with. On top of that, it's approaching unbeatable for single stream performance; HTTP/2 helps with latency on multiple concurrent streams, in exchange for overhead in the single stream case.