RubyGems Fracture Incident Report
27 points by schneems
27 points by schneems
This is a disappointing look for Ruby Central. I have to get back to work, but their retroactive framing that Andre and Samuel's work on RV justified Ruby Central's subsequent actions is contradicted by their own admissions.
By their own admission, André is a contractor to Ruby Central. Contractors, especially under California law, have no contractual obligation of confidentiality to the other party unless there's a pre-existing agreement in place. They later admit in this "incident report" that they didn't have any legal agreements with André in place, so there's no basis for claiming André couldn't work on rv.
Samuel was an employee, not a contractor, but California Bus. & Prof. Code § 16600 voids non-compete agreements—so even as an employee, he had every right to work on a competing project. There's no indication that he used Ruby Central's proprietary information to do so, and the report doesn't allege that. I have little doubt that if Samuel or André used proprietary information to develop rv, they would have already presented evidence of that.
Independent of the legalese, a "uv but for ruby" is a blindingly obvious thing to do, and Ruby Central doesn't get to lick the cookie and get upset when an independent contractor—Ruby Central's own characterization—does a thing they didn't fund.
My sourcing on this is that I run a 10-person business with employees in California. I'm not a lawyer, but I looked over enough of this paperwork that I feel confident opining on an internet forum.
One followup, since this felt extremely dirty: the report insinuates illegal behavior on the part of André and Samuel by selectively juxtaposing facts to imply wrongdoing without ever directly stating that their behavior was illegal. For example:
rubygems-github-backup access token covering "all repos, including private repos" is introduced in the same timeline section as RV development, without any allegation it was used for RV.By my count, Ruby Central makes roughly ten such insinuations throughout the report, but not once do they actually claim any of these constitute a transgression.
insinuates illegal behavior on the part of André and Samuel
I don’t think I’ve done that anywhere in the report. What are you referring to?
My most harsh criticism is that behavior here was "unprofessional" and also (as admitted in my link) I've experienced something similar from the perpetrator side. I don't think I'm being unfair.
not once do they actually claim any of these constitute a transgression
The purpose (again) isn’t to convince you, it’s to paint a picture of what happened. If you saw RV and Ruby central paid offsite together (because they happened on the same day) and alarm bells went off…congrats that the same thing that went off in the heads of the people in the oss committee and the director. If any of my insinuations are wrong or any of my facts misleading I’ve made a public pledge I will update them, recant, and say I’m sorry https://ruby.social/@Schneems/116216591665738708.
I don't claim I am unbiased. I don't claim I didn't make any mistakes. I do claim I'm trying as hard as possible to represent an intellectually and emotionally honest view of what happened. As it happened. As people experienced it. Preferring to use artifacts and timestamps where possible.
do they actually claim any of these constitute a transgression.
Yeah, that's intentional. They could have been honest mistakes or misunderstandings. You are invited to draw different conclusions. Some things that were worries ended up happening so I don't think they were incorrect reads on the situation.
I believe you when you say that the ambiguity is intentional on your part. However, your report doesn't exist in a vacuum, as Ruby Central's incident response post repeatedly refers to André as an "unauthorized actor," while simultaneously noting "we have no evidence to indicate that any RubyGems.org data was copied or retained by unauthorized parties, including Mr. Arko." Concurrently with that blog post, André has noted that Ruby Central's attorney sent his lawyer a letter alleging he had committed a federal crime.
Unless Ruby Central’s legal posture has changed—and I’ve seen no indication to suggest that it has—your report that arranges facts to imply wrongdoing in a way that dovetails with Ruby Central’s present legal stance. I genuinely believe that your intentions are well-meaning, but unfortunately, I can’t extend the same grace to Ruby Central.
Understood now.
A thing my report did is make it clear (via sharing the full text of the email he received) that he knew he wasn't supposed to be there. He knew he wasn't on call and he knew he wasn't supposed to have access.
Legally/privately this is all known. Publically he has misrepresented (generous phrasing) the situation, which is consistent with his behavior in the report timeline. There wouldn't be an implication with sharing those details if Andre's account of events lined up the facts.
Personally, I shared guidelines for holding ME accountable in what I share https://ruby.social/@Schneems/116324218751819984 and how I am framing it.
Unless Ruby Central’s legal posture has changed
It hasn't. From the boards most recent blog post this Sunday:
"Ruby Central did not initiate litigation and has consistently sought a path that would allow the community to move forward without prolonged conflict."
I believe that to be true, still. But also understand it's hard to take my word for an opinion without details.
but unfortunately, I can’t extend the same grace to Ruby Central.
Heard. Ruby Central is 6-10 volunteers in a trench coat (board and oss committee). For better or worse.
A thing my report did is make it clear (via sharing the full text of the email he received) that he knew he wasn't supposed to be there. He knew he wasn't on call and he knew he wasn't supposed to have access.
That is contestable! André mentioned he was the primary on-call and Marty’s communications were erratic (the “I messed up” email, the public GitHub comment promising good faith engagement, and receiving the off-boarding email). The 10 hour gap is explainable by André being shocked before trying to carry out his contractual obligations.
Legally/privately this is all known. Publically he has misrepresented (generous phrasing) the situation, which is consistent with his behavior in the report timeline.
That's a strong claim. Can you point to a specific statement André made publicly that you believe is factually false? Why wasn’t this in the report?
Ruby Central did not initiate litigation and has consistently sought a path that would allow the community to move forward without prolonged conflict.
That’s not what I claimed, and that is a much more specific denial (“initiate litigation”) than what I claimed (“Ruby Central’s attorney sent a letter to André’s attorney alleging a federal crime”). A threat letter through counsel is not initiation of litigation, it’s a pre-litigation demand.
That is contestable!
I believe that he believes his own claim that he was confused about being on call. I also believe that when he reviewed the details to explain his actions, he should have looked at the emails and said, "Oh, actually, this is pretty conclusive. I should say I'm sorry and own the misunderstanding." Rather than choose to distort the text via omission.
If expanding a quote to the full sentence changes the meaning of the usage...it's not a good faith representation of the situation. Going from "terribly sorry" and "I messed up". To:
"I'm terribly sorry about the GitHub removal. I messed up, and I accidentally removed all org access instead of downgrading."
This full quote does not introduce uncertainty (IMO) into either prod access or on-call prior comms from “After consultation with the OSS Committee and the Ruby Central board, we have removed your RubyGems.org production access, given your departure from Ruby Central. We’re also pausing the on-call rotations while we work through this transition. Please send a prorated invoice for on-call services."
Can you point to a specific statement André made publicly that you believe is factually false?
2 weeks ago https://gist.github.com/schneems/577a909b22a2bec2d2b0f06c58711951.
A threat letter through counsel is not initiation of litigation, it’s a pre-litigation demand.
I understand the nuance here, and I think that's the spirit of the wording in the RC board post. This isn't is some hidden "if I omit this one word...it makes my argument magically true via a technicality" scenario.
Some things that were worries ended up happening so I don't think they were incorrect reads on the situation
I think their worries happened because of the actions they took and was not inevitable. It became a self fulfilling prophecy. But who knows what could have happened if they had acted with better communication and less panic.
By worries, to be clear, I meant retribution, and unauthorized server accesses.
The worry of a walk away was also present and you're right I think. I think what you said is true. But from both sides. It takes two sides to have a conflict cycle and neither can stop it on their own.
The wounds here weren’t new. These patterns didn’t appear overnight.
Cross linking. I replied to you on Reddit https://www.reddit.com/r/ruby/comments/1s8of1f/comment/odl906g/?context=3.
As an outsider, something I find confusing about this is (if I’m not misunderstanding):
Please correct me if I’m wrong
That is accurate.
Unsaid in this timeline is that for some unknown reason they were acting with urgency - almost a panic - which seems like why communication was so bad and there were so many unforced errors. From the outside this panic is unexplained, and at times they claimed they had to act fast for legal reasons. It seems to me entirely artificial, and I think they knew it was wrong as they acknowledge they feared a maintainer walkout.
It’s all very disappointing, and while I appreciate this transparency in comparison to their previous communication, it hasn’t changed my opinion that their actions were morally and ethically wrong, probably legally too (but that’s murky and too expensive to litigate). Schneems is trying to do the right thing and repair lost trust, but some of the people actually responsible are still there (Marty and Shan), and I don’t trust them. There’s no revelation here that justifies their actions to me, and I wish there was.
the people actually responsible are still there, and I don’t trust them.
This is probably my closest relevant response https://www.reddit.com/r/ruby/comments/1s8of1f/comment/odkn7r3/?context=3.
Also, I don't fault you for taking what I shared and making your own opinions even if they are different than mine. I also wrote that I warned internally "don't expect people to clap" if this report. (And I appreciate you acknowledging the effort).
The purpose of sharing the hard ugly truth is to learn and grow from it. The harder it is to stare into, the more important it is we do it. The only thing I want to make clear is that it an intentional nakedness. Not a "we were 100% right" or anything. The intent isn't to be "justified" the intent is to be understood. Even if that understanding was mistaken or misguided. Then stare at it and learn from it.
The framing that "Ruby central does not hold this access" is the framing of "the maintainers." It's a statement the Ruby central board was surprised to hear.
Marty did not have access. But the two acting OSS directors before him (Martin and Andre) did. And Evan Phoenix held them on behalf of Ruby central for over a decade prior to that.
This is the kind of sticky situation where having "governance" (real governance with the consent of all the governed including the foundation and Ruby core buy in) is important. There was a last ditch effort there but it was too little too late.
Instead of moving the access control side of things over to something they do control, they opted for attempting to gain control over the existing GitHub thing
Considering they did the maximally invasive thing and still ended up with unwanted access in their servers I don't think it would have been a good idea to do a more complicated thing and hope there are no hiccups. (Opinion, the context is this https://rubycentral.org/news/rubygems-org-aws-root-access-event-september-2025/ which happened outside my timeline but the a ruby dev would known about).
The framing that "Ruby central does not hold this access" is the framing of "the maintainers."
Here's a direct quote from your write-up:
Ruby Central, in turn, wanted to cleanly offboard them and sever ties with RubyGems.org production access, which was tightly coupled to GitHub access. However, Ruby Central lacked the structural ability to make this change directly (did not have admin controls on the GitHub Business/Enterprise)
Correct. Marty (who was full time OSS Director) was never granted admin access by his acting predecessor Martin. Samuel also had it too and was a full time security engineer. As he was offboarded it should have also been a flag that doing so would have left Ruby Central (as a business/foundation) with no structural access controls (through a paid employee).
What I'm saying is "who should hold GitHub access" became the accidental point of contention. It wasn't the intent from the Ruby central perspective. And it is contested. Hence the "walk away" that is what the "walk away" is about.
I adopted the framing of "the maintainers" for the purpose of some parts in a pursuit of fairness. To try to partially explain why they felt the way they did in a way they would (possibly) agree with. But I'm worried it muddies the waters.
Was severing the tight coupling of access controls to the GitHub thing considered?
Yes, but it's an even bigger lift. It still hasn't been done. Its still important, but less urgent now. It was on the roadmap from a prior security audit (before Sept 18). And it's still there. (Source code is public, you can see how access works).
Not to mention, the security engineer we lost would have been the one to take that kind of work. (We are currently hiring that position, paying a good rate. It's a good cause even if there's some [optional] drama that goes with it).
Just want to say that I got a very different take away from this than the current top comment. I appreciate the work that went into investigating it trying to put it together in a cogent timeline and postmortem.