Who wants to crowdsource LLM scans of articles submitted to Lobste.rs?

They have a technical report with methodology.

There are false positives, but they are extremely rare[^1]. Even if you don't trust their marketing materials, they get tagged often enough that we'd see more examples if false positives were common. (There are some! But it's like a single digit number of examples out of many tens or hundreds of thousands of runs.) I'm guessing your understanding comes from an impression "AI detection services" in general, rather than Pangram in particular.

[^1]: as long as you consider "used AI to tweak wording" as a true positive, which they don't distinguish from "entirely written by AI"

I agree. I like writing about the books I read and it's very discouraging to receive comments saying it's ai slop. It sucks the joy out of sth I do purely for connecting with others and make no money out of.

Genuine posts get drowned by the ai posts already, so flagging genuine content as ai discourages creators, leading even less human content, aggravating the problem.

It's difficult to persist in long-form text these days, isn't it? My own blog, already obscure, now exists as a sort of Monument Of Attempt, where I revisit it now and then with an idea for a post, only to be disuaded by how much time and effort it takes to write them for an audience of one (me) in a world where we built potentionally useful tools and instead decided to use them to suck the joy out of any creative endeavor, and now to build a panopticon that rivals anything even the best speculative fiction authors have invented over the years.

Still, I would like to encourag you to keep at it. If you do, I will. We all need to make that promise to each other, I think.

I'm so sorry to hear that... Writing is important to me as well. I hope you can get back to writing once the AI shitstorm passes!

Have you tried running your articles through Pangram 4? I'd have assumed nowadays it's very unlikely that Pangram would flag multiple, full-length, 100% human-written articles as AI-generated.

While Claudish and LLM prose is annoying and frustrating to read to me as much as the next person, I don’t think this creates an environment that people would want to participate in.

I’d be interested to see some statistics and trends of submissions over time. Would be interesting to see any correlations with the addition and removal of the vibecoding tag or various landmark model releases, provided it’s not used as a shaming device.

Vote for what you like that is of a quality you appreciate. Flag where appropriate.

I don't like reading slop either, but I agree that this doesn't feel like the right response. If you want to crowdsource something, why not crowdsource a list of sites that promise to not use LLMs in their writing (and make an extension that gives them a like green dot or something on readers like HN/Lobsters)? You can bootstrap by using a regex like "made/written.*hand/on my own/without (AI/LLM)" on articles.

I feel like rather than conducting witch hunts we should instead give recognition to the people who produce writing we like.

I've seen multiple accounts of "AI writing detectors" (including humans that can "always tell") just encoding biases:

• https://marcusolang.substack.com/p/im-kenyan-i-dont-write-like-chatgpt
• https://kathleenschmidt.substack.com/p/exclusive-author-jerry-falade-reflects

From Jerry Falade's article, I found this persuasive:

The process will be absurd: I pay an AI detector $20 to confirm that I wrote the book I spent several months, perhaps years, writing. It gives me a 95% AI score. I spend months rewriting the book, killing my voice in the process. Then I pay again, and hope the machine is in a better mood. The future is looking bright for writers.

Not only would this mean authors must pay a "Pangram tax" to ensure they aren't falsely accused, but critically I have to imagine people with more money than talent will just game the checkers anyway.

I am not automatically bothered if an LLM was part of the writing process of a document I am reading. I understand that some people are, but, to me, the really upsetting thing is the flood of low-effort, uninteresting spam that we have had to endure since people started using LLMs to write. Putting a number on the amount of LLM use on an article does not resolve this issue.

I am not exactly sure why there are so many LLM spam articles. Personally, I do involve LLMs in my writing process, albeit only sparsely to correct grammar, say, or to help me phrase a section I am struggling with. When I write something it's because I want to communicate my thoughts or my work, so, fundamentally, I do not understand why people publish slop articles.

This makes me think that the slop articles we are all bothered by are just written in bad faith; to gain visibility, out of vanity, or as an advertisement. This has existed before AI, too, but I guess it wasn't such a problem because people actually had to spend time to write. To me, the problem is the bad faith part, not the LLM part. Checking the amount of LLM use is a bad proxy for detecting bad faith.

I really don't know how to deal with this situation, though. To be honest, there existed plenty of articles in the era before LLMs whose style of writing and presentation I found insufferable. Take for instance the recent sshole page (https://sshhole.com/). All of it reads like this:

so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead. that character can never match a real password, so programs don't crash, they just see a wrong password

now guess which character they picked for that "*"

yep. the star. again.

This style of writing was a thing long before LLMs. It also seems like the content on the sshole page was written by a person. Still, I would appreciate it if things like this were not on here.

I am bringing up this example, because this style of writing makes me feel disrespected, just like LLM slop articles do. I would like to see a taste filter or a care filter, but I have no idea how to implement such a thing. Link aggregators like Hacker News and Lobste.rs served this purpose for me in the past, but it's not working as well anymore. Maybe the reason is not actually that people can now use LLMs to write, but, rather, that people have discovered these link aggregators as a great way to boost their profile in an incredibly competitive job market. Or something along those lines.

Is there an OSS effort to replicate Pangram?

This is a solved problem imo - if you get the sense that it's AI generated, and it's usually pretty obvious when it is, flag it as spam and move on.

The biggest concern I have with LLM scanners is that they penalized any use of LLMs in the past. This includes people who feel like they need them to improve their writing, or who use them to translate their writing (I understand both of these motivations even though I personally make different decisions). Have these issues been solved?

We use to be able to look at a certain level of quality of grammar and punctuation as a proxy for "this probably had enough effort put into it to be worth my time," and I get wanting that back. I don't think it's been a great heuristic for years, though. Think of how much useful information is in a poorly formatted forum post and how little is in a glossy, well edited corporate white paper. LLMs have made the latter widely accessible.

I'd be more interested in an LLM tool that tries estimate how useful the content is. As a starting point: how concrete and widespread its jargon is, does it have concrete problem statements, does it provide advice or solutions in a form that bounds when they are useful and what you can reasonably expect from them.

Hi Adolfo, I'd be interested in this project. I agree; there is no sense in multiple people scanning the same article. I'll send you an email. Cheers.

I understand that a lot of people here dislike the idea of using AI for writing, and I respect that. But at the same time, I think there's an important distinction to highlight. Saying that something was written by AI isn't really the same as saying that something was written with AI assistance. There are many shades between those two extremes.

I admit I'm someone who uses AI to correct and aid my writing. Maybe it gives me a tiny bit of extra confidence as someone who is not a native English speaker/writer, and in general, I like to believe it makes my writing better. Still, I put a significant amount of effort into making sure that what I write is mine, that it conveys my ideas, and that it does so in a way that reflects how I would have expressed myself. AI, hopefully, just makes it more correct from an English perspective and overall more readable.

Fun fact: I just checked my latest article (which no human that I know of has complained about so far in terms of feeling like slop) against Pangram, and it says it's 87% AI, which seems way too unfair, almost depressing, to me.

I wrote most of the article myself and then went through various iterations with LLMs to improve it until I was happy with the result. It took me more than a day of work overall to produce that article, and I like to think it provides value and isn't slop... But hey, that's my opinion, of course, and I'm surely biased.

The point I'm trying to make is that I personally believe we're chasing the wrong things here if we're simply afraid of AI adoption. I agree we should fight slop and, in general, low-quality content. And in that sense, I also like to think that community-based systems of comments and upvotes already help the best content emerge naturally.

If we just want to fight the use of AI in general, that's an entirely different battle, one I wouldn't necessarily subscribe to (it's debatable for sure). Sure, bringing something like Pangram into the picture could help with that, but it doesn't necessarily mean we'd end up with only high-quality content just because we're filtering out anything that seems AI-written to some degree.

My two cents...

No - this feels like the sort of witchunt that @pushcx regularly pushes back against in moderation.

Not sure how it is effective: https://humanjson.org/

I takes a completely different approach, though.

Either the article is worthy content or it's not, let upvote mechanism determine what counts as good writing or not.