Rolling Our Own S3
39 points by veqq
39 points by veqq
Trimmed the spam title, which the author even hates.
My first thought after just barely skimming the article is...
yeah, it sounds like S3 was never the right tool for this job.
It’s not clear even EC2 is the right tool for the job. My instinct is you could colo a few servers to do this whole pipeline a lot cheaper.
EC2 is almost never the right tool if you care about cost
And one almost always does care about cost. (Just some managements consider it a given, they cannot get around)
Did at least scalability side get any better over the last few years? No more apologetic emails coming back, about how they are working very hard to increase capacity at some location when you actually do wanna scale up quickly and try to increase your limits? At some point this was the reason we had to switch away. Of course the realization that your cloud provider can't scale comes at the worst possible time. To be fair: Similar things do (maybe still?) happen at Google cloud.
So running a bit short on use cases for using the big cloud providers.
Fun fact, if you dig a bit or look into Wikipedia's version history: One big selling point of cloud providers was reduced costs and those were explicitly linked to coming from multi-tenancy and having virtual systems so you can make much better use of your resources. And because of the economics of scale you can massively reduce the costs even compared to ordinary vservers. I guess there is a reason why Amazon doesn't know where to put its money. But certainly doesn't seem to be reasonably priced services, reliability or salaries of people in logistics.
With all the talk about breaking up of Google I think Amazon should really be the one to look at. Even only taking the part they pay taxes for their income is at a level that maybe shouldn't be that concentrated like that.
Cost reduction story for cloud was always "pay for what you use" that is, turn off servers you aren't using. In practise people ended up using them for web servers though, which need to be always on and you can't start them fast enough to absorb load spikes either so everyone just overprovisions and runs forever just like with classic hardware...
the non-nagware version: https://scribe.rip/how-we-saved-500-000-per-year-by-rolling-our-own-s3-6caec1ee1143
Been a dedicated Nanit customer since my kid was born (2 years ago). Almost all our parent friends we've met (dozens) all have a Nanit, too. It is one of the very few things I tell new parents "must buy, no complaints."
I can't recall a single time that I've noticed Nanit being unavailable, and as an engineer I've definitely been wary of that (we do have a non-wifi backup camera we use primarily for hotels when traveling). Nanit is my most opened app across my Apple devices (via Screen Time). I don't hover over my kid's sleep, I just have it open in the background constantly primarily to listen to audio. I've done this for hours daily for 2 years without stop and it has never been down for me.
So, they're doing things right in terms of end user perception (taking off my engineering hat).
Regardless of the content here, I'm just happy to read about some of the inner workings of this device. :)
Curious about what’s so good about it in your view. Seems like a normal baby monitor with some AI stuff built in or something?
Hi Mitchell, author here, thank you for the kind words, I've shared it with our engineering team and we really appreciate it and excited to have you as a user! :)
I'd be tempted to have tried, as a quick first step, to make the video service enqueue requests to delete the objects from S3 as soon as it was done with them rather than waiting for the automatic cleanup rules to get them, just to cut down the storage costs.
That wouldn't have helped the PutObject costs, though.
You're absolutely right, we actually did that before going all-in and replacing S3. But for our use case, the storage costs were just ~20%, and most of the cost was from the PutObject requests, so we continued optimizing.
This is a cool solution. My first thought, however, is why not use MinIO to replace S3? This seems like it would be cheaper to implement than the custom solution. Perhaps overall operational costs might be higher, depending on the resources consumed by each.
MinIO has been AGPLv3 since April 2021 and I would presume that dissuades most commercial vendors. There are plenty of other S3 API compatible self-hosted stacks, although I'm positive they carry their own tradeoffs, like any technology
The AGPL/Free Software "community edition" of MinIO is either dead or in the process of being killed, depending on how you interpret this comment.
As with so many other instances, AGPL was just a moat to prevent commercial competitors while the primary SaaS business developed proprietary features, and I for one wish the "Free Software" community would stop falling for this.
While termination of releases for the freedomware edition would be bad, it's still better by far that it exist than not since we can keep using it and maintaining it ourselves if they drop it.
Or the FSF could develop a new AGPL-alike that adds a single clause -- all contributions made by downstream recipients are received by upstream under the terms of the AGPL and no other terms, thus putting upstream and downstream on equal license footing -- and watch it rot on the vine, thus proving that the AGPL's de facto purpose is to enable the creation of proprietary software using copyright to enforce a monopoly on commercial exploitation.
I don't think such a provision would be enforceable but I'm not a lawyer and would love to see a draft.
That said, lots of AGPL software has no proprietary anything and like I said, even for Ines that do the AGPL is very useful in terms of being able to keep using it as free software forever.
AGPL provisions only trigger if you modify the code. Just running the binary comes with no special requirements. And also they sell a license to those who choose not to understand this IIRC
Nice post. I'm still amazed whenever I see people working only on AWS primitives, feels like cloud brainrot (of course I have bare metal brainrot instead)