Creepy crawlies
47 points by gmem
47 points by gmem
FWIW, git hosts have always interacted very poorly with crawlers, to the point where you have to actively code in git host detection to avoid getting stuck in an accidental crawler trap if you want to run a well behaved crawler. Easiest is just to look for anything that looks like a commit hash in a path and drop those URLs from the crawl frontier.
Reason they interact so poorly is that is that they generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host.
Do we know what fraction of this traffic is from (IMO) malicious SDKs embedded within apps vs. bot nets?
I feel like those two categories require different (ideally upstream) solutions. For example, it seems plausible that app stores (which claim to protect users) could be forced to ban residential proxy SDKs. For bot nets, I guess ISPs might be best positioned to (be forced to) block compromised systems. Note: these are just examples that sprang to mind!
I expect that the residential proxies don’t have enough compute to be efficient going through Anubis, but I don’t think they care much. If it takes many seconds to load, whatever, it is not THEIR compute.
The compute part doesn't matter for the proxy. It's the requesting node that will have to solve it; the proxy is just passing the bytes. The hacked consumer device is not the crawler node itself.
They don't need to have much power at all - only bandwidth, that's why they're proxies! The compute part doesn't need to run on the proxy - the proxy only needs to, well, proxy. A bog standard headless chrome in a datacenter can - and does - happily go through a residental proxy.
I'm very unconvinced this is actually AI. There's a bunch of bots pretending to be AI crawlers going around. For an AI model, you really don't want to train on licensed code a thousand times repeatedly, it ruins your generalization. What's actually behind it, no clue though.
This did not happen 3 years ago. I'm not sure if it's for training itself, but it's somehow related. And they're not selective - this kind of traffic is basically stuck iterating on really deep links and the operator doesn't seem to care because the cost of mistakes seem to be close to 0.
What's happened in recent years in parallel to AI is that residential proxy industry has exploded in size and capability, which changes incentives as you don't have to worry about burning IPs like you did before the business model took off. As long as you yourself don't care about the web as an ecosystem, there is no longer any incentive to throttle your requests.
There is a new industry of data companies who do this, not the ML model training companies themselves.
And then some ML model companies buy the datasets from those, with that level of indirection.
As a result, a lot more market players crawling and building their datasets in the hope they'll find a buyer.
It's odd that this is still a secret today. As it is widespread, I would have expected some blog post "how my company is trying to scrape the whole web against its will."
I'm dealing with a similar thing at work and it's really annoying. The kernel site is in a worse situation though where they can't run good behaviour analytics, because it's likely someone genuinely comes to the site straight from some external deep link to a decade old commit.
Some ideas that could work for them: (or anyone else dealing with this)
Scale the difficulty: Static website gets lower than a commit, which gets lower than a custom diff. Frequently requested commit gets lower than an unknown one. If it's not too hard to get the age of a commit pre-check, maybe even vary on that too.
Get some fingerprint analysis going. Some crawler networks work around that, but a huge number does not. I won't spell out the details, but... just put things on a dashboard together with ja3/ja4 and you'll see more patterns. Then you can bump the difficulty again (or block) in more targeted ways.
Identify some definitely-good traffic that gets lower difficulty. At the extreme, no crawler pretends to be Gnome Web running on Arch ;)
Start dropping links to other pages for questionable cases. "If you're a real user, visit the front page to reenable navigation links".
The only actual scalable solution is to serve static content only and render everything client-side. Want a diff between two arbitrary commits? Looks like you're going to have to download both and render it yourself.
Of course this sucks since now most of your content requires javascript, takes too long to load, and your network bill went up.
This is likely not scalable. To get two specific commits you're doing an equivalent of two shallow clones, because you don't know what files you need to compare. At that point you may as well just remove the option on the website because nobody is waiting for that.
The only actual scalable solution is to serve static content only and render everything client-side.
That helps nothing when your bottleneck is TLS handshake. When you find yourself at the wrong end of a 10k+ request/second scraper wave that lasts days, no matter how efficiently you serve static files, the TLS handshake will kill most cheap VPSes.
By cheap, I mean I scaled my normally €4/month VPS at Hetzner up to like €40+/month during a particularly nasty wave, and that fell over too. Not even static files, 2 bytes ("OK") + HTTP response headers served from RAM.
My solution ended up being firewalling them off. Which I could, even residential proxies, because they're dumb.
It's ironic that they complain about kernel.org getting so much spam traffic while basically all the slop corporations are members of the Linux Foundation.