good-egg: Trust scoring for GitHub PR authors based on contribution history
5 points by a5rocks
5 points by a5rocks
It's important that we have ways to combat PR spam caused by LLMs. But I don't think this is a good solution, particularly because it can be gamed so easily. Create a few LLM agent accounts and make them make and merge PRs for a bit. Then the merged PR ratio improves and the contributor is given a "good" ranking.
Unfortunately, I think we're going to need to have human-based vouching systems for contributors or find other ways to increase friction for LLMs in particular.
I also find this text in the repo's CLAUDE.md ironic considering a large part of the problem is non-disclosure of LLM-generated code.
No AI attribution: Do not add Co-Authored-By, Signed-off-by, or any other trailer attributing AI/Claude to commits. Do not sign PR descriptions, comments, or code comments as Claude or any AI assistant. Commits should be attributed solely to the human author.
I share your concern. Maybe the list of top-contribution repos can mitigate this? If the repos with the highest contributions are completely unknown or even clearly LLM slop, that could raise concern even if the merge rate is high?
Maybe the trust-score should depend on the viewer: weight the contribution by the "distance" of the PR with my own contributions.
So that PRs made to projects where I also contributed/commented/starred gets taken into account, while projects unknown to me gets ignored.
Create a few LLM agent accounts and make them make and merge PRs for a bit
I don't think good-egg tries to oppose genuinely "bad actors". It seems to target "lazy" first-time contributors
So if my contributions were made outside of GitHub, I basically do not exist, as far as this tool is concerned. Not that I'd want to contribute to a project hosted on GitHub, but one using this tool, I definitely would not.
Being hostile to drive-by contributors is a sure way to not have any drive-by contributors.
I haven't used this nor do I know whether it's any good. The idea sounds nice though; I've found slop PRs to feel worse to review than I expected, and having some proxy for effort-put-in that I can see at a glance would be good.