Offensive Internet Posture
47 points by zk
47 points by zk
- Some user agents (Meta’s) always get 403 HTTP status codes.
- Some user agents (xAI’s, Claude, GPTbot among them) get sent to an infinite fake website.
I wonder why the different strategies here.
This website’s contact page does nothing but waste spammers time and effort. It also sets between 0 and 25 randomly-named cookies, so maybe don’t access it.
That one might be a bit too much, I have to say. The contact page is even linked on the site's nav bar.
I'm the author. Meta does not do rate limiting well, so the number of requests from Meta went up and up and up. The same was true of Bytespider, which I believe is Bytedance's (TikTok) crawler. I'm running a $6.47 a month VPS, it couldn't handle all of Meta and all of Bytedance. Claude and GPTbot seem to do rate limiting of some sort on the same fake data.
I haven't had a legit contact form response ever. I don't think people (Real Humans) use them. I have had a number of direct emails about various posts.
I've used contact forms, but only ever to reach people/orgs that didnt publish an email address.
Nice. On my own sites I serve all requests by obvious bots (like those probing for php files or .git directories) an excruciatingly slow 1 byte/sec redirect to https://blog.cloudflare.com/cleaning-up-bad-bots/. I'd like to do more, maybe send an endless zip bomb, or at least something that is more likely to tie up more resources than just the connection state.
Given the prototype for gzread is
int gzread (gzFile file, voidp buf, unsigned int len);
you have to actually go out of your way for an internet crawler to succumb to a zip bomb in any meaningful fashion.
They may feel good to set up, but to be honest most of these countermeasure tactics don't even register for a crawler. The web is super adversarial as it is and any of these countermeasures you set up is already encountered on a regular basis and handled. The by far best thing you can do to make impolite scrapers go away is just serve a default nginx index.html or some such. Then it looks like your server is up, but there's nothing to find, and mostly they go away.
you have to actually go out of your way for an internet crawler to succumb to a zip bomb in any meaningful fashion.
From that function signature, are you saying that gzread reads into a buffer of predetermined size? So if it processes a zip bomb, it will prematurely stop decompressing after only len bytes?
My naive assumption is there must be a way to request continued decompression, and that a lot of software would do that by default. Even if the C interface is bounded, I can imagine Python repeatedly calling a C function and copying the contents of buf into an ever-growing object.
I do agree that these countermeasures probably aren't that effective. I just haven't heard this particular argument before for why zip bombs don't work.
Yeah, exactly. gzread reads into a bounded destination buffer, and most higher level implementations offer a wrapper that exposes a "get N bytes"-type method which is what crawlers generally reach for.
But even without explicit gzread calls, crawlers generally avoid holding data in memory (as they can have many thousands of ongoing requests in flight at the same time, you end up with insane malloc thrashing), instead preferring to shovel bytes via small bounded buffers into temporary files as they come in.
And! Regardless of whether you're reading a gzip bomb or a particularly large Linux ISO in this fashion, you need both a time-limit and a size-limit, and a SO_RCVTIMEO limit, and a per-domain time and document limit. You end up with mitigations for most of these attacks by the fact that you need to deal with regular web data.
I just realized who I was talking to, and it makes sense that you would know a ton about web crawling. :-)
Maybe the most you could say about zip bombs, etc, is that they might trip up poorly-coded scrapers that don't impose resource limits. But like you said, a lot of real-world web content will force you to do that anyway—so one would expect that any such hypothetical scraper will become immune to these attacks shortly after being put into service.
Oh interesting. I do a bunch of stuff with streaming html and I was wondering how bots would interact with it. Basically, I send the next frame of html every 50ms over a brotli compressed SSE stream. If there's no or minimal changes that can be 13bytes over the wire but 130kb uncompressed per frame. Fine for browsers because they are only holding on to the latest frame. But my guess is it would be bad for a bot.
Hard to say. Crawlers generally run on fairly low tech stacks, like HTTP 1.1 and gzip compression at best. You generally don't see any benefit from request multiplexing or any of that stuff in this type of application, and it creates issues with pooled connections in mystery states after an aborted request and so on.
Might be a good congruence test for fake browsers that pretend to be real ones though. If you claim to be e.g. chrome, you should support more than HTTP/1.1 and Content-Encoding: gzip.
I understand the reasoning behind all of these countermeasures. The net sure is in a different place today than it was when I was running rwhod(8) on my hosts.
I do run a telnet tarpit of sorts on nugget.info.