"Proof of use" as a requirement for open source contributions?
34 points by twp
34 points by twp
As an open source maintainer, I get a lot of AI slop PRs. Some of these have genuine value. Most address edge cases that almost never occur in real life. It's not that they're not fixing real problems, it's that they're fixing problems that nobody is ever likely to encounter.
I'm experimenting with "proof of use" as a requirement for submissions. Specifically, you need to show me that your project that depends on my project before I'll look at your PR.
"Proof of use" can be an existing open source project that imports mine (show me the git history!) or a clear description starting with "in our closed source project in production, we found that...".
The specific problem that I want to address is ensuring that contributions from people genuinely depending on my software are handled, while filtering out the OpenClaw/Github green square farming/drive-by LLM noise.
What are lobste.rs's thoughts on this?
Alternative way to reduce green square farming is to not be on GitHub (or only be there as a read-only mirror), but rather on Codeberg or another less-mainstream more community-values-oriented host. But that sounds like a fine thing to put in CONTRIBUTING.md, sure.
Codeberg definitely feels like a safer space, but there's the pressure to host issues and more on github for ease of collaboration :(
I felt this preasure for a ling time, but I don't think it realky applies anymore. Like what are you afraid of? The cryptobros will tell you you aren't cool because you host your open source code on an open source forge?
I do have to say I moved my active projects to codeberg. But I keep the github mirrors because github is the only place where you're going to get issues and PRs and discovery. It's not the cryptobros, but just people in general are still mostly still on github at this stage. Of course it's the same phenomena that keeps Meta going. Hopefully the other places will get critical mass, because github is falling apart.
Codeberg forbids LLM-assisted contributions whether they are valuable or not.
That is not what their TOS says:
You must not share projects that mostly consist of code written by "generative AI"-tools (including services such as Claude, OpenAI Codex). Such projects having an unclear copyright status (see requirements § 2 (1) 1 and § 2 (1) 3) and furthermore have little safeguards to ensure that they do not include harmful code (c.f. § 2 (1) 5).
"mostly consist" is quite vague, but the introductory blogpost does clarify things better and it seems to be largely aimed at majority/entirely vibecoded apps.
In fact, if you consider the phrasing carefully, "code written by" doesn't imply you cannot use LLMs for research/debugging/ideating/etc., they're just requesting you not to sling code that no human eyes ever saw and exhaust their already tight resources, that they'd much rather provide to people who actually need it.
I mean it's your project. But if I use it and report a bug, and even go to the lengths of providing a patch and it gets rejected with this, chances are good you'll never see me again except complaining about your project.
Of course this might be worth it. But I'm relatively sure you'll remove most of the "trying to give something back" people. I operate under the premise of "Here's my code, hope it is useful to you, patches mostly welcome" and so I also contribute back like that.
To clarify, I've never provided an LLM patch, but I'm obviously biased if my contributions were useful. I'm not denying in any way that they may be drive-by, it's been a while that I fully associated with a project and did regular "maintenance" on it.
Well, as a maintainer, I will then be glad not to have merged your patch.
Because merging the patch is more than often not about the code written. As a maintainer, it would probably takes me less time to implement your patch myself the way I want it than to review your patch (let’s be honest, most first time contribution are often trivial).
When I receive a patch from a new contributor, I always start by asking questions about the patch: why did you do that? What was the purpose? Why not this way? (except, of course, for very trivial patch)
The point is not the code, the point is to see if I’m facing someone reactive or willing to engage or someone who would hit and run.
Because even if your patch took you hours or even days of work, I will have to live with it afterward for years and hopefully decades.
If I’m accepting your patch, it’s in the hope that you will contribute more and more in the project until a point where it would save me time (and that point requires quite a lot of contribution). This will only happen if I’ve learned to know you, to trust you. It takes an awful lot of time. And it happens very rarely. But it’s the only way I see to build a healthy community.
People who are not long term maintainers of Open Source don’t realize that your patch is NOT a gift from you. It means more time from me, often at a point where I was planning to do something else. If often conflicts with my own priorities for the project. But I see it as a lottery ticket that may bring me a long term collaborator. If the first frustration keeps you out, then it’s better to know in advance.
I had an example where I received a very big patch for a feature I find really useful. There was one line in that patch that didn’t make sense to me. I send that feedback three times. Twice, I received as answer a new version of the patch which didn’t address that specific line. The third time, I didn’t receive any answer at all.
Well it probably means that I cannot trust this person. We can’t communicate. So it’s better to know that before engaging in a deeper collaboration.
You don't have to explain that to me, I've been in that place. Maybe I was just lucky that I hardly had to reject such patches. There are patches that are easily merged and basically free of side effects. Don't act like most people would submit a diatribe of "oh and btw, I unrelatedly replaced mysql with postgres".
Rereading what I wrote maybe you took offense at the "and complain" part. Rest assured I have never complained just because my patch was rejected. I have complained if the maintainer behaved like an asshole while rejecting a (useful) patch or just messed up in some other way. My go-to example was nearly 20 years ago in a time before github where you had to collaborate with a maintainer to get your patch submitted (from a mailing list, or whatever) and someone dragged their feet for so many weeks that someone else (kinda independently) reinvented my patch and committed it (because they were a maintainer) - that's just lost time and effort for 3 people.
This ^^^. But even more - if a maintainer skips something important to you then that's a signal.[Details redacted because the dev doesn't owe me anything.]
Last week I had Claude analyse an OSS security project. The main developer also works on the closed source variant. I had a lot of reservations about the OSS code quality and Claude found a vuln. Something could realistically only be exploited if a step was scripted instead of manual as the product implicitly expects.
I submitted the explanation and one-line patch. My goal was to evaluate if the maintainers share my level of care about security.
Result: silence. So I decided to not use either the OSS or proprietary products.
This wouldn't work for me. I do not care how the contributor uses my project and wouldn't want to spend time reviewing or verifying "use proofs", which could be easy to forge using LLMs. It may also appear to be a barrier of entry for someone who wants to contribute but does not have a project to show.
More than a proof of use, what I would love to be able to implement is a proof of "interest", because I only want to work with contributors that are interested in their seeing their contribution through. Those who just want to improve their numbers I have no problem turning down. But I haven't figured out how to test for interest.
As others have suggested, I have moved all my projects off of GitHub to a self-hosted Forgejo instance, where only people I give accounts can contribute. This is actually what solved the problem for me. The repos on GitHub are now following the main Forgejo repos. They have the option to submit PRs disabled, so nobody can bother me there anymore. I still allow people to write issues there, though. I consider these issues as drafts that I need to review. Any that I find valuable I transfer to the real repo on my Forgejo instance.
In my case, I’ve moved to Sourcehut, where everybody is welcome to contribute with an email.
But sourcehut itself has strong protections against bot which means very little to no slop patches. I’ve expermented by trying to get an AI send a patch to my own project and it failed because it could not access the source code at sourcehut (it spent 30 minutes looking for way to circumvent that without success. I don’t say it is not possible but it seems good enough to avoid most slop)
I will piggyback your thread to add that SourceHut allows setting repositories (mailing lists, trackers etc.) as Unlisted. So, one can have a public repository but an unlisted patch list, and then simply share the list link to private groups or individuals they want to be able to contribute.
I self-host Darc repositories with a HACKING.rst where emailing a patch or pull request is the way to send code & all collab is on the XMPP MUC (yes, I have received patches, PRs, & some chatter). This certainly filters out a lot of stuff—probably humans too, but we really should own our code & decentralize out chat.
Any system needing proof adds a verification burden to yourself. It might be less work to rely purely on your taste.
I don't know if this is just turtles-all-the-way-down, but couldn't AI help you triage it?
I do this on closed source (I call them cold reads) and it's proven quite effective.
I think this is a decent idea, yeah. When I file issues against upstream projects from $work I try to include clear cases where we actually hit an issue.
"proof" is a strong word, but I do ask for confirmation of connection from those submittors who otherwise appear to be farming green squares.
So now you don't just have to review a vibecoded fix, you also have to review their vibecoded app. No thanks. :D
I'm experimenting with "proof of use" as a requirement for submissions
Ah, it's a classic "Is there ROI" discussion that happens in large tech teams, too. Is it worth putting in effort fixing minor bugs and edge cases? Only here it's all about people throwing LLM code at the author.
I think it's rather reasonable to request the user be invested. Any change comes with a burden, and a change from a stranger has extra burdens (especially with recent surges of supply-chain attacks) of validating that stranger. Requiring a short description of their situation will let you quickly review whether they're a slop-machine slopping away, or a real user. IMHO, burdening them to unburden yourself is the right call.
The people this will drive away were probably never investing much more than a 5-10 minutes vibe coding session to "contribute" anyways. Not much of value was lost.