The Decline of Search Engines is an Opportunity
81 points by LAC-Tech
81 points by LAC-Tech
One thing I unfortunately noticed is that Google actually puts some of the best links in the AI overview panel, and they usually are better than the top results.
I just googled "what is a linked list" for reference a the side panel inside the AI section shows fa Reddit thread on top, and then the corresponding Wikipedia article (not great, I still think Wikipedia should be #1, but, anyway...).
The top search result is a GeeksForGeeks article, and I mean no shade on GeeksForGeeks but it is very SEO'd and it ranks very high despite better content being available on the web. Wikipedia is also #2 in this case.
Even if LLMs do a better job at picking the damn links (and I don't think they do, this must be carefully designed), this is a prime example of enshitiffication. They nest some good links in <div id="slop"> and, if you click them, the metrics will count you as a happy slop consumer, because at this point you know the actual search results will be dogwater anyway.
I mean no shade on GeeksForGeeks
By all means shade on that damn website. It is very good at SEO and rarely actually has the information I am looking for. It was slop before the word slop became widespread. It is one of the websites I have blocked in my search results.
I just googled "what is a linked list" for reference a the side panel inside the AI section shows fa Reddit thread on top, and then the corresponding Wikipedia article (not great, I still think Wikipedia should be #1, but, anyway...).
For me the biggest revelation was to minimize my use of "general-purpose" search engines in cases where there was obviously a better more focused engine I could use. Like why bother doing a google search for linked list when you already know that the wikipedia page is likely to have what you want?
In Firefox I've added shortcuts so @w linked list would go straight there. There are plenty of other more focused, specialized searches you can do, like the Arch wiki, MDN, Stack Overflow, IMDB, etc.
Honestly the whole idea that one company can be trusted to operate a service that finds any piece of information in the world is very suspect to begin with. I wrote some about this last year:
It's tempting to wax nostalgic about how things used to be. "It was amazing," you might say, "you could just put a few keywords into Google, and if it existed, you'd be able to find it!" The problem is that was never true. In exchange for saving us from a chaotic web where you bounced around and couldn't find what you wanted, we allowed Google to change our very definition of what the Web was. "If it's not in Google, it doesn't exist," we said. Of course, we were wrong. But we're only now starting to see how wrong.
The problem, of course, is that this objective gods-eye view of the Web fundamentally cannot exist. Indexing something as big as the Web while resisting abuse requires making thousands of judgment calls; some large and some small. Whenever you search—no, whenever you abstract—you must accept a certain simplification of reality. Just as the map is not the territory, the index is not the Web.
There's the &udm=14 self-titled "disenshittification Konami code", which always redirects to the "Web" search tab (normally hidden under "More"). I've enabled it across my devices a while back and been very happy to have a "dumb" search back. Still has all the flaws of SEO and ranking gaming, but it gives me back some agency that I feel like the AI overview stole.
I mostly use DuckDuckGo, but thanks a lot! I went to check their bangs list for something that uses this flag and they don’t have it yet. They have a bang for something that they call “Google Accessible” but it uses Google’s programmable search engine and I’m not sure if this maintained by the DDG team of if some user just decided to submit it as a suggestion. It looks much better, though. I’ll investigate, but, for now, I’ll keep away from it.
This has been happening for a while (I blogged an example last year) and it is extremely annoying and must be deliberate so that people stay on Google sites longer.
I consider the bargain between Google and web sites almost null and void - if they are not prepared to show links to my site, why should I allow their bot to spider my site?
As mention in your blog, if you did not yet, just try out Kagi. It is well known that they are using the crawled data / search index of another large search engine. But only the data and they are doing their own ranking for the search result.
I am using it already for more then 2 years, and I am happy with it, I did even find stuff where DuckDuckGo failed. And there are a lot of more features like to raise, lower or block sites from your search result or doing Redirects (URL Rewrites). What I also like about Kagi is their open communication, like a public Changelog with links for each ticket out of Kagi Feedback (which also functions as their issue tracker) and their Status page. Regarding your idea doing a link page on the own site, I guess the Kagi Small Web is a somehow similar idea, visit a random site at Kagi Small Web website.
Even John Gruber (Daring Fireball) and Cory Doctorow (Pluralistic) are using it, see e.g.:
Another Periodic Suggestion to Try, Just Try, Switching to Kagi for Search
The Talk Show: ‘A Professional Internet User’, With Kagi Founder and CEO Vlad Prelovac
Tag: kagi (pluralistic.net)
There is a Trial with 100 searches, and depending on your search amount you may get away with the Starter for $54 / year (annually paid) or higher depending on your needs, see Individual Pricing overview. They have also offers for Duo and Family which then would be even less per person, but only available as Professional. You can upgrade or downgrade at any time; your unused balance will be pro-rated and applied to your new plan.
But Kagi is using Yandex, so paying them is indirectly supporting Putin’s regime. I don’t want to knowingly support killing Ukrainians.
I agree that we should not support Russia's war of aggression against Ukraine. I disagree that it's possible to use a search engine without supporting atrocities. To quote from the UN Human Rights Council, "Alphabet, Amazon, Microsoft and IBM [are] central to Israel’s surveillance apparatus and the ongoing Gaza destruction." I am not aware of a commonly used general-purpose search engine + index that doesn't use data from one of those companies (shoutout marginalia, though!). Brave doesn't qualify as an ethical search engine, for obvious reasons.
I disagree with a lot of Kagi's decisions and their stance on AI, but I don't know how I would improve their search engine. They do build their own index and run their own crawler, which must be even harder now that everyone is locking their sites down against the flood of scrapers.
I disagree that it's possible to use a search engine without supporting atrocities.
I don't think that merely using data from those companies is supporting them. If you use Google but have a proper adblocker like uBlock Origin, you are costing them money and not driving revenue. If you use SearXNG and it pulls data from Google, you are even further removed from supporting Google. The data isn't the issue, it's the revenue.
Your argument for not avoiding Yandex, is that other companies are no saints?
Presumably you avoid all the companies mentioned in their response, in their entirety?
Actually I do avoid all of them and for sure I’m not paying them any money directly.
Bro u have a GitHub link in ur profile, that's Microsoft
This "gotcha" nonsense is foolish.
It's about minimizing your dependencies, not stopping using everything.
Sure. But it's quite rich to call somebody else out for using a product because a portion of its revenue is channeled to unethical firms, whilst simultaneously using other products that ultimately do the same.
Pointing out the unethical link is one thing, but the snarky followup was very unnecessary.
Otherwise, I absolutely agree with you that reduction over absolutism is the right move for most people.
I said I avoid them, not that I’m perfect. Just like I didn’t delete my personal GitHub account, I also have inactive old gmail account.
This has nothing to do with the fact that paying for kagi is transferring 2% to Yandex.
You also buy absolutely no Nestle product, or.. well is there even an ethical company after a certain size? I would wager that it's pretty much impossible, being "ethical" is just good PR, so some companies opt for that.
Oh, I was not aware of that, and so I did some research. In the blog posting Waiting for dawn in search: Search index, Google rulings and impact on Kagi (January 21, 2026) they do mention, that they also have licensed from Yandex, but not with Google and Bing and so are not able to use their data directly and mention this: "Because direct licensing isn't available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results (SERP meaning search engine results page)."
As Yandex was also not mention in the link given from @diktomat, I looked in their Feedback forum and found Reconsider Yandex integration due to the geopolitical status quo (Nov 20, 2024) with an answer from Vlad (the founder) and there seems to be only a very small amount going to them. But sure, I also would like to be able to exclude them for my searches completely.
Yet, after 2022, DuckDuckGo terminated it’s relationship with Yandex instead of pumping more money into Russia.
It is well known that they are using the crawled data / search index of another large search engine.
Actually, they’re aggregating multiple sources. I agree they’re great though, just renewed my subscription.
What's it like compared to old Google when it gave good results?
I think the search results are about the same quality in a vacuum compared to Google ~3 years back (when I switched), but for me the killer feature is being able to set priority for individual domains. This allows me to block SEO slop pages and increase the visibility of pages that I know are high-quality.
In my opinion Kagi is what Google Search used to be at the beginning. Around 15+ years ago Google did give personalized results based on your previous searches (even without login). I had switched to DuckDuckGo around 10+ years ago, but it started also to decline.
1: DuckDuckGo, to my understanding, does let you block sites from results. I don't think it lets you uprank or downrank sites, or at least I never tried.
2: It should be noted that Kagi was started as an AI company before pivoting to search [https://help.kagi.com/kagi/ai/kagi-ai.html]. That is not to suggest anything in particular, but I don't know how many people were aware of this so I am sharing that.
More directly, I suggest others to be wary of your advice since you seemed to phrase your entire comment as an ad.
ddg's "block sites from results" has a limit of like 10 websites or so (don't remember the exact number) which makes it effectively useless for accumulating a large blocklist of slop websites
Glad you bring this up. Wasn't aware of this limit and now I am quite annoyed. Hoping they can fix this or another provider can step up. It's (probably) not hard to implement...
Regarding 2:, I may have seen this before I started to use Kagi, and in some cases I did like their "Quick Summary", which is using AI with the content of the first few results. In my opinion a good use of AI, as it is not generating new content but just a summary, with references from which result it is.
It was not my intention to sound like an ad, I am just a happy customer so far, and I like to support alternatives where I am the user of a (paid) service and not the product.
I also use Kagi and it's the best $5 I spend. I spend most of my day searching for things and dealing with Google has become harder and harder. Duckduckgo, bing, are just not good enough in many cases. Kagi is OK.
I do hope they don't add AI to their product and make it worse, like everyone else seems to be doing nowadays. So far, so good.
I liked Kagi, but stopped using it when I learned they're pouring money into Russia.
Recently I've picked up Uruky, and I gotta say I like it a lot. It's not perfect, but better than what google has become. I haven't dug too much into how they source their data though (just verifying it's not Yandex), so I'd be interested if anyone knows more.
I've heard a lot of praise for Kagi, but I haven't had the same luck with it. That's not to deny your experience — I rather suspect that I've been "holding it wrong". So I'm curious as to what kind of searches that you've been putting into Kagi?
The benchmark I've been using was "sesans spin echo length formula". Under DuckDuckGo, the first link is the Wiki page for SESANS, which doesn't contain the formula I'm looking for, but the second link is a paper that has the formula. With Google, the first link is a conference proceeding with a lot of the mathematical basis for SESANS, but not the formula I'm looking for. Again, the second link has a paper with the formula (and a better explanation of the constants that the second link under DDG). Under Kagi, it wasn't until the sixth link that I found the relation I was looking for.
So, under my benchmark, Kagi is doing worse, but I suspect that it's just a bad benchmark. What sort of searches should I be doing to see Kagi at its best?
I have tested Duck Duck Go, Startpage and Kagi extensively, on many different search terms.
On average, Kagi is the best, and on average Duck Duck Go is terrible, at least for the kinds of searches I do. Startpage is quite decent, better than DDG, and like DDG is can be used anonymously over TOR.
Kagi is not always the best. Sometimes I get zero results in cases where Startpage will find a few results. In these same cases, DDG just gives me an avalanche of garbage.
DDG prioritizes showing me a large number of hits over showing me the specific niche thing I am looking for. If I search for a very niche thing using the correct spelling, in quotes, then DDG will instead show me a large number of hits for a popular thing with a slightly different spelling. I assume this behaviour is driven by doing whatever produces the most clicks and ad revenue.
afaik Kagi searches aren't logged (at least officially) so the anonymity shouldn't matter, but they also offer Kagi privacy pass which is an extension to let you perform searches with Kagi's TOR service (link) without logging in.
Startpage is quite decent, better than DDG, and like DDG is can be used anonymously over TOR.
I just accidentally used Startpage this morning, and if there were 10 results on my screen, 9 were sponsored. I thought it was horrifying.
Oh, in that case, I should clarify that I use uBlock Origin, so I am not seeing these ads.
Ah! The way I came to accidentally use it was that I popped open a private browsing window to see what a site I was working looked like to a not-logged-in user, then forgot the window was special and did a web search. For whatever reason, my browser defaults to startpage for web search in private browsing mode (and none of my blockers are enabled by default in that mode), and I was horrified that only one result, at the very bottom of my screen, was not "sponsored". Then I looked to see if something had changed my settings and realized I was in a private browsing window.
I am mostly searching for IT technical stuff, and sometimes for everyday things in my area. I did had some issues (error messages from my IMAP server) where back at the time DuckDuckGo was not able to give me useful results. When I started to use Kagi, I did the same search (line out of the logfile) again and Kagi did find the corresponding bug report, which pointed me to the real issue why it happens, and I could fix it. The relevant information was already in the bug report when I had searched with DuckDuckGo.
I just tried your sesans spin echo length formula search, this is a shared search result, so see for yourself. For me the summary and first link is Wikipedia, and the second link is pointing to "Spin-echo small-angle neutron scattering for multiscale structure analysis of food materials" at www.sciencedirect.com.
Those were the same links that I found when I did the search. While both of them introduce spin-echo small-angle neutron scattering, neither of them give the specific formula for calculating the spin echo length. I needed to go the [https://webster.ncnr.nist.gov/programs/sans/pdf/Seminar20090304.pdf](the sixth link) to find the formula I was looking for in equation one on page 7.
I tried now your search query like this: "sesans spin echo length formula?" The ? at the end does give a Quick Answer on top. But unfortunately it does not work with the shared link. It gives some details (with annotations with links to the source) and a formula which looks like this (as copy & past did not properly work, I had to "create" it somehow manually with 'pre text', but I guess you will understand):
Mathematically, it is often expressed as:
γmλ2
δ = ----- ∫ Bdl
2πh
Where:
δ is the spin-echo length.
λ is the neutron wavelength.
γ is the gyromagnetic ratio of the neutron.
∫Bdl represents the integrated magnetic field strength (the field integral) experienced by the neutron along its path.
I am not a mathematician, and don't understand what I am looking at, but based on your link it looks not like what is there on page 7. It really seams that you are searching in a very special area, but giving Kagi Feedback may help to improve their results.
And what happens when Kagi turns to slop? Where are we then?
No one needed Kagi before google turned.
I believe you that Kagi is probably a great alternative, but I disagree that it's any kind of real solution.
Just trying to follow links is also not an option. It worked great when the web was still small and more of a place to spend some leisure time. In today's world, the amount of information is far too much to digest all by yourself, even if you were to remove all the SEO crap.
IMHO both are necessary.
The whole point of what I am saying is that most information online is shit and not worth digesting. The only way i can see to convey trust and legitimacy to other content is if people (or organisations or whatever) you trust link to it deliberately.
At least you have the choice to be the user and not the product.
And if we look at the history, Kagi is around since 2018 with a slow grow in team size (currently 50 people), with probably compared to other Silicon Valley startups with very little investor money, but a constant grow in users subscriptions. See About, History and Stats for more details. Also the The real cost of "free" search and following pages may be of interest.
On the other hand Google Search was free to use, and the user was the product. According to The Age of PageRank is Over, Google in 1998 did mention that they don't like ads, but just a few years later they started to turned into the now biggest online ad business.
So decide for yourself.
I've wondered if a human-curated directory, like Yahoo's original site, is the way to go these days. That, plus a way to search those sites. I think marginalia is somewhat like this, but I don't see a directory.
I had the same thought. The author has basically reinvented the web directory. Which probably needs to have its day again despite the scalability question.
I don't participate here much so my apologies if this reads as shameless self promotion, but I've been working on a tool like this and it's been in alpha testing stage for a couple months now.
It's a tool for creating collaborative web directories: https://linklists.net
It's rough around the edges and is a bit of an art project in addition to being an indie web tool, but I'm kinda proud of it. It's born from a love of the linked era of the web and my desire to help keep it populated and easy to navigate/explore.
A while ago there was Where can I find the old internet? posted, with the comment regarding Wander Console which is similar to https://linklists.net mention from @yharnam in this thread.
Kagi is often recommended but AFAICT relies on the 3 providers mentioned above.
I don't know if this is true but it literally doesn't matter. Kagi search is absolutely brilliant and I'd probably still pay for it if it cost double. I make on average 1,432 searches a month, totally worth it.
when you ran kagi and other search engines side by side what were the differences that you saw? any sample searches?
I don't know if targeted, specific side-by-side comparisons really matter when you can change your keywords a little and get new results. If you're running a statistical analysis for best search results instead of going by what actually works for how you search in practice, I don't think you'll ever be happy.
Think of it like this: Startpage didn't satisfy me. Duckduckgo couldn't contain me. Google is, simply put, beneath my search prowess. The only search engine that satiated my lust for search results is Kagi, simply demonstrated by the fact I continue to pay for it instead of using a free search engine.
I saw two comments mentioning it but it deserves to be linked: Marginalia Search is great!
In 2024 Google and Reddit made a $60 million deal to use Reddit content for Google AI training. For this reason, doing a Google search today is like using site:Reddit as prefix. Good old websites are hidden behind irrelevant and low quality results.
The degradation of search engine capability is fairly undeniable at this point.
Am I the only one who has not noticed this? I still search for things and find them...
I think of basically persistent caching everything and indexing for full text search. Especially since some things seem to fall out of web or I just can't find them. Projects like that were shared here before. My basic idea is SQLite with FTS5 for everything. Need to explore this space.
I built my own hyper-personalized search engine with sqlite FTS5. I don't have it indexing my browser history yet; just my bookmarks and every URL I've posted to my social media account. But I've been thinking about it for sure. https://search.technomancy.us
And I highly encourage folks to experiment!
Oh, yes! I remembered someone did something like this and thanks to your comment I could easily find the previous post I've seen here. For the curious: why I built a hyper-personalized search engine that isn't very good
Thanks for inspiring me to finally pull the trigger and publish my own links section: https://shime.sh/www/. I was overthinking it.
I posted a response to someone else in this thread but I figured I'd talk about it at the top level too as I think it's relevant to the blog post. As I said in the comment though, if this is too self-promotey or rubs anyone the wrong way I'm happy to remove it, just lmk.
I'll try to keep it short and light on the sentimentality, but I went through a period of mourning what I thought was the loss of the web I grew up with, largely because I didn't know that many parts of it are still out there. The modern web of course isn't exactly the same, and a lot what I'm "mourning" is nostalgia, the age that I was, etc., but despite that the web is still huge and full of interesting, useful, fun things. It's just really hard to find it now, even before AI tools started eating search engines.
I thought a lot about what experience I was missing and for various reasons came to the same conclusions as the blog author. We need real humans to share link to the good stuff, and we need to foster a slower, more intentional web-surfing attitude. In that spirit I made a tool for creating collaborative web directories: https://linklists.net.
It's tiny and it has a lot of rough edges. It's kind of an art project as much as it's a tool, so I've done very little promotion and really just spread it around with some indie web folks. It's been in alpha for a couple of months and I already have some big changes I think I need to make, but if anyone is curious and wants to give it a shot, check it out. :)