An update on the scraper situation
98 points by fanf
98 points by fanf
Out of all the openly shady crap I have learned about recently, "residential proxies" seem to be the one that stands out to me the most.
I read more about them at https://spur.us/blog/smart-tv-apps-residential-proxy-sdks
I can't believe they are legal, they should be classified under the same category as botnet, in fact they should be called legalized botnets.
This stuff is just reprehensible. They talk about vetted/authorised partners, but I cannot think of one legitimate usecase for a residential proxy. Is there one?
Avoiding censorship and accessing content that's gated only to a subset of rDNS and ASNs. If these were any cheaper people would be using them to watch Youtube.
archival and good-faith scraping
these are indeed good causes.
it still requires treating your subject as an adversary. with archivists scraping a big tech hosted site, they have our sympathies; but being on the other side simply feels unpleasant. and that's probably why everyone in this thread (including me) has such a visceral reaction to these kinds of proxies.
Both of those kinds of clients normally access web sites directly; legitimate archivers and scrapers do not need residential proxies. If a scraper needs to use a residential proxy it is almost by definition acting in bad faith.
The use-case i believe lyra had in mind (or at least, which she reminded me of) is guerilla archival, like https://archiveteam.org does. but even they i think simply run their tools on their own home networks and rely on many active collaborators to get around rate limits.
Except that works well enough with volunteers; without requiring commercial residential proxy services and embedding malware in TVs
I'd be careful cracking down on residential proxies. The deploying of the botnets without consent should be illegal, but the users using the proxies should also not be discriminated against. This is more of a circular problem created by heavily relying on WAFs and if the person unknowingly is running a proxy their traffic ends up getting blocked as well. I am growing extremely weary of security companies and large cloud providers making their internet equivalent to the war on drugs.
In principle I agree, but then what's the remedy? I think as an operator it's reasonable (though not desirable, clearly) to tell a user "you've installed some malware, and you need to remove it before you can access this site because your device is making the internet impossible to operate." That assumes, however, that the block and block reason is clearly communicated to the user, which I'm sure it often isn't (much less malware removal instructions).
I don't think this is the optimal solution overall but I also don't think it's unreasonable to do as an operator without control or influence over e.g. browsers, operating systems, or app stores.
Another way to think about this: if a website could reliably detect credit card stealing malware on a user's computer, would you make the same argument? If a physical toy store (I picked this because it isn't an essential service) could reliably detect that someone about to walk into the store was infected with a disease, would you make the same argument?
Another side victim are link checkers. It becomes quite difficult to check if your site has dead links or not. If I had time, I would look at the Common Crawl project to check for content liveness.
Other side victim: sender-generated link previews in chat apps. I was happy to contribute it to gajim (an XMPP client), but in practice, the link preview hits "checking that you are a human" so often that it's depressing.
link preview hits "checking that you are a human"
holy crap
I didn't take a look at anything like anubis&co, but to me it sounds like they should at least "forward" opengraph meta tags to make link previews usable (I don't have og set up, but I don't check for bots client-side either)
update: it seems it's possible to do something like I said and it is a default (source)
I didn't take a look at anything like anubis&co, but to me it sounds like they should at least "forward" opengraph meta tags to make link previews usable
That would result in highly increased server load or resource use, because reaching back to the backend to grab the OG tags is not free. Caching them isn't, either. You'd also have to let requests for the images in the OG meta tags through, too, further increasing the load, and also exposing part of your content to the crawlers.
The end state will be that all expensive operations (e.g. list a github repo history) will be protected behind an authentication barrier, and everything else made static and served over CDNs.
I think CDNs ought to be a last resort (due to my own concerns around centralization, government surveillance, etc) and that we ought to try to wring more performance out of our servers instead.
At the application level, making more things static definitely helps, like you said. Reducing database queries helps. Caching helps. For a long time we've had the luxury of not having to worry too much about performance, but now we have to think about it more, even for small sites. And maybe that's okay? It's much better than Cloudflare "verifying your browser" everywhere.
Anecdotally, I've run into problems with Apache on small VPSes that were due to the more old-fashioned approach of forking separate processes for each request it handles. I grew up tinkering with LAMP servers, so Apache is very familiar and comforting to me. But it's not making the most efficient use of the machine, and at the end of the day that's more important. I don't have empirical evidence for this, but I suspect something like Caddy would hold the load better, and it's what I'll probably switch to the next time I upgrade my server.
I've run into problems with Apache on small VPSes that were due to the more old-fashioned approach of forking separate processes for each request it handles.
Apache hasn’t forked per request since before I started using it 30 years ago. It traditionally (in 1.x) pre-forked and kept a pool of processes ready to accept connections. Since 2.0 the default is to have multiple threads per process, and in more recent versions there’s an “event” mpm which avoids needing a whole thread for simple things like writing a response body to a socket.
Oh interesting. So it's more correct to say that each incoming request didn't necessarily trigger a fork (due to the pool), but that it still relied on there being a process that had previously forked. And maybe I had something poorly configured, because at least in my case it seemed like each request occupied an entire process, and the number of processes in the pool was a bottleneck when serving bursts of traffic.
Checking my server right now, it looks like I am using prefork, but somehow I'm not getting the multiple-threads-per-process that you mention. Again, maybe something is misconfigured on my end. I remember seeing bursts of requests where I had 5 threads in the thread pool, 5+ requests rapidly came in, and latency spiked after the 5th one because all processes were occupied and it had to grow the pool.
I'm not doing anything unusual: this is stock Apache on Ubuntu LTS (22.04). It's good to know I can probably squeeze more out of it. But it's a shame that it's this easy to get a subpar Apache config, and if I'm having this experience, others probably are as well.
You want the MPM worker model (multiple processes, each with multiple threads supporting requests). You'll also want to disable (comment out) any Apache modules you don't use (reduces foot print and reduces the number of checks per request). One module that doesn't come with Apache but has helped me is mod_limitipconn which throttles the number of concurrent connections per IP address. I was amazed when I realized there were bad bots out there connecting hundreds of times per second from the same IP address.
You want the MPM worker model
Digging into this more, I think I know why MPM is disabled for me: it's because mod_php is not thread safe, and installing that forces Apache to use the older prefork style. If you apt install php as part of a basic Ubuntu LAMP stack, that's what you get by default—the less performant one.
(PHP-FPM is the modern alternative. Sounds like it's also limited to one request per process, but in a separate process pool so Apache can serve non-PHP requests faster. Sorry @fanf for badmouthing Apache! I meant to badmouth PHP… j/k)
tl;dr: Defaults still suck, and it's still worth trying to optimize our servers before we surrender to the CDNs!
Sorry @fanf for badmouthing Apache! I meant to badmouth PHP… j/k
Heh, well to be fair I didn’t mention the many caveats with Apache’s more advanced mpms. Later web servers generally learned from the mistakes of Apache / mod_php / mod_perl and booted out application server stuff to separate fastcgi processes or the like, so they could be decoupled from directly dealing with all the wild and woolly clients out on the big bad internet. (With the odd exception of nginx + Lua in OpenResty…)
I switched to nginx years ago, but you should be able to set up Apache to prefork workers and use the sendfile system call to serve static content. Caddy is nice for it's auto TLS feature but can be bloated if you install a prepackaged distribution rather than compile it yourself.
Automatic support for Let's Encrypt was the other half of the reason I was looking at Caddy! I'm not sure I like their plugin philosophy ("just recompile Caddy"), but when I was last playing around with it, it was so much easier to get a basic HTTPS site up and running.
Regarding bloat: do prepackaged distributions include a lot of plugins by default or something? Curious if it makes a perf difference.
...or we'll use smarter detection.
Case in point: you can visit any of my sites, like my linktree. You will most likely see it. No auth barrier, no CDN. It is static, but that doesn't make much of a difference here.
Yet, ~90% of requests against my sites get served garbage (90% of those that get through the firewall - a lot of the residential proxies end up firewalled off in the first place).
With a firewall up, and automatically banning the worst offenders for 12 hours resulted in my sites seeing ~2.5 million requests a day (down from about 100 million). The CPU, RAM, and bandwidth costs of this is minimal, a tiny fraction of what a CDN would cost, and I have no auth barrier, and I didn't have to re-architect half my stuff to be mostly static either.
On the other hand git repos interact extremely poorly with crawlers, to the point where any half-competent one will actively avoid visiting URLs with long hexadecimal strings in their path[1], as they create a sort of ad hoc crawler trap with their combinatoric explosion of dynamically generated links.
[1] As much as this is extremely expensive for the git host, an odd number of git hosts don't set up a robots.txt to avoid deep crawls into git repos.
Yeah that is my situation as well. I tried to manage consistent abuse on my Forgejo server with Anubis but I didn't fully understand how to tune its settings to address the kind of traffic I was seeing, and I wasn't really a big fan of being presented a slow authentication challenge myself.
I also tried to use fail2ban but that too didn't quite scale, and ultimately I had to simply let Cloudflare handle it through a set of security rules that I documented in my blog post. I was rather hesitant of relying on Cloudflare long term, but the CPU usage graph you see at the bottom ended up convincing me.
There have been requests to describe the measures we have taken to defend the site; for obvious reasons we do not wish to discuss them in any detail. It is an arms race at this level too.
this pains me, too. i've been fairly successful at this game myself, but the tragedy of the commons means talking publicly about the really simple measures i've taken makes them less effective. but this topic is really interesting and i think i have some unique ideas which i'd love to write a blog post about :(
Recently, LWN was subjected what was, by far, the heaviest scraper attack yet. Thanks to the defenses that have been implemented, the site bore the traffic well enough that most actual readers probably did not even notice. There have been requests to describe the measures we have taken to defend the site; for obvious reasons we do not wish to discuss them in any detail. It is an arms race at this level too.
If you have access to logged-in traffic --- and, even better, logged in-traffic associated with actual money involved --- then I would assume that that traffic is your gold standard against which all other traffic patterns are compared.
Though frankly, I'm astonished that even websites as spartan as LWN are struggling to handle scraping traffic. Just how much traffic has to hit you at once in order to make it impossible to even serve plain HTML and CSS? Wasn't the C10K problem (and its solutions) meant to make scraping, even foolish and poorly designed scraping, effectively background noise compared to human traffic?
If nothing else, it fascinates the old system administrator in me. I haven't been in those trenches since 2014, and clearly, the game has changed.
Just how much traffic has to hit you at once in order to make it impossible to even serve plain HTML and CSS? Wasn't the C10K problem (and its solutions) meant to make scraping, even foolish and poorly designed scraping, effectively background noise compared to human traffic?
Things have become way more expensive since C10K. My €5/month VPS can trivially serve 10k/sec on HTTP. Add TLS into the mix, and over HTTPS it falls over at around 1k req/sec - TLS is very expensive.
On my own server (the aforementioned €5/month VPS, 2 CPU cores + 4GiB RAM), when I was hit by an 1k req/sec wave, it fell over due to the TLS handshake. Nothing touched the disk or anything else, just the TLS handshake made it fall over and die. When I scaled it up, the crawlers scaled up too, and at one point I had 20k req/sec incoming, and I couldn't afford to scale the server up higher (eventually ended up implementing automatic firewalling against the worst offenders, and now I have a constant hum of 60 req/sec 24/7, which is nothing, but considering its almost all crawler traffic, still insanely stupid).
And that's on my niche sites that don't appear in search results, and have a normal human traffic of maybe a dozen visits a day (on a particularly good day!). Now imagine the bullshit LWN has to deal with, if small sites like mine see this kind of traffic.
When 99% of the traffic is scraping, that's never going to be background noise, no matter what you do.
I wonder if you could get some of that performance back by turning off support for RSA and DSA, allowing only elliptic curve cipher suites.
Seeing these recurring stories of even niche blogs getting hammered, I'm wondering what I'm doing right (or wrong?) that this isn't happening to my websites. That is, I do the following:
The end result is that I'm left with somewhere around 10 000 requests per day, and about 5 000 per day if you only count unique IPs. Pretty much all of that traffic is feed readers that are too stupid to not send a request every 5 minutes, though most of them are at least nice enough to perform conditional requests.
Maybe bots just hate my website? Or maybe nuking Alibaba and Tencent really does help.
While logged-in traffic may be the "gold standard" it doesn't necessarily match logged-out behaviour. And while it is most important to keep logged-in users happy if they cut off too many anonymous users they may find less people signing up because that traffic is effectively a marketing cost. (Ignoring the goodwill and moral benefits of providing free access to simplify the argument.)
While logged-in traffic may be the "gold standard" it doesn't necessarily match logged-out behaviour.
That difference is, presumably, what makes the "human vs bot" question answerable. I'd assume that humans, whether logged in or not, are more alike in their patterns than they are with bots. But alas, it's an arms race, so nobody is going to be spilling the beans on what actually does and doesn't work for a while.
Is there a summary of the different ways of discouraging bots and encouraging legitimate traffic, while keeping financial sustainability? I'd like to see them categorized by stuff like:
If we take a paywall like NYTimes as an example, it requires a lot of infrastructure, but seems to still attract hundreds of thousands of users. Not sure if the signup is accessible, it probably requires a lot of JavaScript to integrate credit cards.
On the other hand, one could make a Lightning paywall for a Gemini site, lose 99% of users, but theoretically still serve content at full speed. Software would be so niche it's hard to say whether it would be accessible or not.
One could also serve all content at 28.8 kbps, stick to pure HTML and aggressive caching, retain accessibility but lose goodies like images, which would become unbearably slow. I like this one but not everyone is into retrocomputing.
I wrote about some approaches a while ago, from the perspective of someone hosting niche things on cheap infra.
For what I host, the solution I ended up with (see the aforementioned post), a €4-5/month VPS to front it works fine, and it is mostly idle. Bandwidth is well within my hoster's limits, and I could make it smaller if I really wanted to. There's no obfuscation for the legit visitor, and should work fine in a bandwidth-constrained environment too. Legit users rarely notice the defenses even existing - except that the sites I host are still up.
Hardly any visitors lost, bots are taken care of, didn't cost a fortune either, and no centralized "solution" like Cloudflare was used.
@meithecatte we were arguing about the value of the proof of work as an anti-scraper system but it seems I missed the depressingly obvious problem: if you’re stealing the compute resources then the proof of work is entirely irrelevant. (It took me so long to find that thread to get your name again :D )