How are you dealing with all the negative side effects of AI on the web?
28 points by tmcb
28 points by tmcb
It seems to me that the AI hype killed the web for good, but not for the reasons we immediately associate with it.
Search engines now rank links full of slop at the top, which ironically make LLM summaries acceptable, not because they are better than what we had before (they are not), but because they at least mention the good links which are now buried at the bottom of the page. Also, they've been rate limiting ordinary users like crazy: more often than I'd like to, Google and DuckDuckGo now either show me either an error message or tell me to wait a bit. Sometimes I waste at least a minute refreshing the page and waiting for a result.
I also can't browse GitHub freely without being logged in. Anubis is now everywhere, occasionally blocking me. I feel like my productivity has actually gone down thanks to all of that.
Edit: I see that most of the replies here are suggesting relying less on the web and going after more fundamental resources, which are great suggestions, don’t get me wrong! But a book from 5 years ago will not help you understand why your custom firmware doesn’t boot in your 1-year old mobile chipset. A existing bug report on the web probably will. Not quite my predicament at the moment, but something very similar for which the web seemed to work just fine before the AI craze.
I'm still working on Marginalia Search. It's gotten pretty good, but still much to go. Kinda wish more people saw this all as a fun and interesting engineering problem rather than just moping about it.
I think AI gets a lot of blame for a state of disrepair that was already quite present even before AI came around. It's exacerbated a lot of the existing problems and added a few new ones, but even in 2021-2023 the web was a monumental shit hole with barely functioning search and discovery capabilities. Astroturfing pre clawd-era was a bit more limited as you had to be a state actor or employ rooms full of people in low income countries to do it, but let's not pretend it wasn't already a big problem.
It's more of a dark forest now than it was back then, in a combination of increased bot activity and the dead sea effect of people who give a fuck about quality conversation being the most sensitive to decline in quality and leaving first. Finding actual high quality human interaction on the web has never been harder.
I think that's the biggest unsolved problem we're facing now. All the solutions for solving the bot problem seem to revolve around dystopian slippery slope tech like device attestation and tying digital identity to physical identity. Not at all happy with that one.
Thank you! For Marginalia (exactly the kind of walk-the-walk we need), and for avoiding defeatism.
Yeah the attestation and identity mechanisms, enforced by state legislation, point to the end state (predicted by net neutrality activists)[https://www.lukekehoe.com/wesupportnetneutrality.html]: internet access akin to cable tv: you pay $N for facebook+netflix, $M for facebook+netflix+youtube, etc, etc. Plus, what you can see and read is highly regulated and monitored.
Personally, I think the small web will need to somehow fork itself off of the mainstream www. I don't know how that will happen, and whether overlay networks will play a role, but we're looking at a speciation process for websites.
There is a Taoist proverb along the lines of: The crooked tree grows, while the straight tree is cut down for wood. I think the small web, the personal web, and also the web of reference docs and historical documents needs to become that crooked tree.
I love your work on Marginalia btw, it's a fantastic resource.
Finding actual high quality human interaction on the web has never been harder.
Indeed. It's gotten harder and harder over the years; big tech and social media ruined so much of it.
And yet I interact with humans online all day most days. Here for one example
Yes, I don't think I or the comment OP were saying otherwise, only that it's harder to have "actual high quality human interaction on the web".
It's the occasion of relying more on communities like Lobsters or on RSS feeds to discover new things.
I agree, but sometimes I don't need to discover new things, I need to check boring stuff that no one else is discussing to get things done.
I don't want to go on a rant but basically it seems AI killed programming as a hobby to me for reasons more than one. The experience of trying to build something myself has become both very expensive if I'm dealing with buying hardware or very frustrating when it comes to do the research I need to write the software. All of that makes vibecoding way more tempting and, to be honest, at that point I'd rather pick a new hobby as I already spend my workweek doing basically the same just to pay my bills. Bird watching is not rate limited.
Gemini is free and it’s a better search engine than Google is. As of December 2025, I find it to be reliable — more reliable than Google was
You don’t have to vibe code with it — you can just ask simple questions and do the work yourself, as you always did.
That said, I agree that AI is systemically bad for the web, and the current situation seems unsustainable. But if you just want to get work done, I find LLMs to be a big jump in capability. I also pay for Claude because for non-search queries it’s less sycophantic than Gemini.
I’ll grant that there is a temptation to use it as more than a search engine, and the results are jagged. Sometimes it goes way beyond expectations, but sometimes it leads to rat holes. Like recommending obscure products that aren’t really available anymore. That happened with plain search too, but it can be more wild with LLMs.
Agreed but I'm very afraid that online communities where you interact with other people and not bots (like reddit used to be and lobsters still is, as far as I am aware) will become scarce.
I feel like I was prepared for this AIrmageddon of the Web by going more and more offline.
I’ve lot of physical books. I’ve relearned the trade (and the joy) of going through books to find information I’m looking for. If I realize a field is important enough for me, I buy second-hand books about the subject from before 2020.
I rely more and more on official documentation and manpages.
When using a library, I have more and more the reflex of checking under the hood and read the source code. It might seems daunting but, once you pass that fear, it’s extremely efficient. You don’t rely on outdated documentation: you see what you library is doing. In some case, this even allowed me to ditch the library for my own implementation when I realized that what I trully needed was a couple hundred lines of Python.
For everything for which I’ve no book, no documentation, no man page, I rely mostly on Wikipedia.
All those 4 points require an investment. Instead of having the right git command from stackoverflow, I skim through the official git book. It takes me 5 minutes instead of 1. It’s harder. But then, I realized something interesting: I had to go less and less for the book. I’m really building my skill, I’m learning stuff for which I was blindly relying on SO, thinking I understood stuff.
Of course, not everything is offline, sadly. So:
I follow a lot of interesting blogs through RSS in Offpunk (to make them offline).
My only social network is Mastodon where the vast majority is against slop. I may also hang out of some good old forums/communities.
I pay for Kagi as my main search engine. In the setting I’ve ask to remove IA slop. In Kagi, you can also give high/low priority to domains. I block completely domain with slops or deprioritize them. I use multiple "block slop" list in my ublock filters. If a website appears in my Kagi search but is blocked as AI slop, I ask Kagi to never show me result from this website anymore.
This is far from perfect, it requires effort but this makes me a lot happier in my own personal life. This requires some sacrifices.
The main problem is that it makes me live a life so different from the others that, at some point, you feel like you are not living on the same planet.
I think that SO was never the right answer to a question and was always a shortcut. Looking into the sources or high-quality documentation was and is the way to go. So for the truly disciplined among us, almost nothing changed.
So for the truly disciplined among us, almost nothing changed.
Maybe “disciplined” is not the right word. Most of the replies here refer in one way or another to offline (as in “Internet access not required”) documentation, which is all good, but there is so much more than that. Some hobbyist projects relied heavily on the web before, and reading old e-mail threads, bug reports or PRs is really helpful to understand them.
Everyone is a little different of course, but I think we should build a new Internet.
This internet's core technology is designed in a way that centralizes content hosting costs. The only way to take back the Internet is to redesign it in a way that distributes content hosting costs, the same way we do in a torrent network.
We should also be sharing the costs of UI maintenance.
It's a shame zero net never took off (https://github.com/HelloZeroNet/ZeroNet). It was basically that: decentralized domains through Namecoin and mostly static only websites that are distributed through users. No centralized servers. No centralized domains.
I try to focus on a positive angle: the web was starting to feel awfully "corporate" and soulless even before this AI wave. The awesome dirty divebar or hole-in-the-wall taco place from the 90s and 00s seemed drowned out by soulless malls attempting to replicate the same feel. Maybe the age of slop will motivate geeky communities to reinvent ourselves. And I don't just mean replacing Twitter with Mastodon or Reddit with Lobsters (I do like this place), I mean really reinventing – new networks built on top of both the internet and mesh networks, where new spaces can form. Protocols not deliberated on by committees and implemented by megacorps, but rather ad-hoc "garage" style development. Spaces that don't get swamped by "ordinary people" and the corporations that chase them (I don't at all mean to gatekeep – at least not on purpose – I simply long for organically grown communities of likeminded people on top of truly decentralized infrastructure).
I was a few years too young to be on the "right" side of eternal September, but I think the web has long ago had its similar moment. It's time for something new. From the grassroots. There's an advantage in that connectivity is ubiquitous and cheap now. I have hope.
Mostly just depending on old PDFs. It's wild what the internet doesn't know (and never knew) about logic and mathematics.
Where are the PDFs from? In my experience, logic and mathematics are fairly well digitized, compared to very different topics such as German poetry.
For publicly indexable stuff, I have found marginalia search to be decent. It doesn't have as comprehensive an index as Google does (providers like Kagi and Uruky resort to paying third party firms to scrape them), but the dev does a much, much better job at filtering out SEO nonsense from oneuptime and the like.
Unfortunately, the closest thing to "grep for the web" for finding PRs / bug reports in my experience is Google verbatim mode. I am very worried about how much longer it will remain usable, and am gathering as many bookmarks (to project forums and docs for example) while I still can.
See also: https://lobste.rs/s/xg8zbk/clankers_made_me_build_second_brain
I've bought an ebook reader, removed most of the apps from my iphone/ipad, currently removing most of my information and data online and visiting way less websites. I use a JS shield which means many websites fail to load for me because of their tracking. Currently, I deactivate it only for websites that I need (basically government/banking sites).
I've decided (maybe also because I am getting older?) that I am taking zero shit from anybody. Of course, I have to play by the government rules. But apart from that, taking no shit. So any service that's giving me a hard time, I am giving it a hard pass.
I think it's been broken, and I don't see a real fix.
I rely more on offline docs, reading code, and I use less online information.
As someone who’s been chronically online almost my whole life. The things you’ve pointed out have actually made me peruse more offline activities.
DDG has an option to disable slop summaries.
How many searches are you doing that DDG shows you a rate limiting screen? I've used it basically exclusively for as long as I can remember and I don't think I've ever seen a rate limit screen.
DDG is going the slop way of Google. One thing I learned is that these services are only "nice" to get their initial users. Once they start to get traction, they all converge into the same destiny. If the founders can't stomach the enshitiffication, they sell to the highest bidder to do the misdeeds for them.
Only way out is decentralization where user data can't be extracted and thus there is no financial reward.
Yes, I just commented about the "crooked tree not getting chopped down for wood" in a comment above, but you put it very well here. Decentralization is the way out.
DDG has an option to disable slop summaries.
I do have it disabled, but there are a couple of caveats:
How many searches are you doing that DDG shows you a rate limiting screen?
So DDG doesn't exactly give me a rate limiting screen but I often get those "Oops" pages, at least once or twice a day. This was never the case to me before 2025. I can't help but think this is all due to bot (btw agents are bots) traffic.
Have you tried kagi.com? They recently made a free tier giving you 30 searches without signup. I only just started playing with it myself, so I can't say if it's actually better than DDG.
Kagi also tags the AI generated results (not all, but close), so they're easy to filter/skip. Also, if you're not opposed to the idea itself, agents dealing with search are way better than both the search summaries and the old search. With the summaries you're getting the bottom of the barrel quality. With your own agent in research mode, you'll get precise sources for many queries that would take a very long time to review otherwise.
I often do my research on private browsing windows when I'm too lost, so the option cannot be enforced
Same and for that I use the url query settings.
My browser search is configured to a custom DDG URL that includes my settings, so I get the exact same experience whether in private or not.
From a privacy standpoint it's not great but since most of my searches come from my residential IP, I don't think it makes it difference to my traceability.
https://duckduckgo.com/duckduckgo-help-pages/settings/params
And the main settings have a "show bookmarklet" option so you can just use their UI to build the right query args, and look at the right incantation for your existing settings.
There are options, but they're not easy to find.
I'd suggest you look for search engines that do not perpetuate the ad-funded business model, but rather where you pay to access a product that puts you front and center.
I only know of 3: Kagi, MetaGer, and Uruky, which I've founded with my wife and launched earlier this year exactly because we felt we needed an alternative to exist.
With all of this said, I believe we really need more initiatives like EUSP (and Mojeek and Marginalia, though at different scales) which are really trying to compete with the index size of Google and Bing. Uruky allows you to use all of those and we have our own "indie/small web" index being built and it's incredibly costly to do it right, and while the big tech bots don't do it right, it's even more costly to go after them.
I hope it's possible to fight back Google/Bing, but it won't be with "the same thing" they offer. I think it'll be by having multiple alternatives with different focuses, appealing to different kinds of uses and users.
Your headline/title is a bit ambiguous: I immediately jumped to the conclusion that replies would talk about how they're salting the earth with fake, infinite, generated websites full of gibberish, or using that weird string that makes Claude stop reading, and hopefully some other spiteful ways to create mayhem for corporations I'd never heard of. This is all good discussion though.
The biggest point of frustration I have is actually Anubis/cloudflare "bot detection". It is in front of so many websites now, and sometimes with ridiculous parameters (there was one site I visited recently with Anubis that would have easily taken 10-15 seconds to solve the challenge). If i click a link that pops that up, it's easily 50% that I just click the back button and go away. I don't care enough about the information to be subjected to that.
It used to be the case that websites cared about how quickly they could get their page in front of your eyes, to the point that that was used in Google search rankings, and now it seems like we've thrown all that out the window.
I get that AI scrapers are a huge problem for website operators but I really wish there was a solution that didn't involve treating real users like criminals.