Distributed, high-performance blob storage on SQLite

27 points by gowthamgts


sebastien

I think SQLite's positioning is that it does not compete with other DBs, but competes with the filesystem. I realised that first hand building an event sourcing system with event batches and compaction. A month worth of simulated data would take up 5-10x the size on the FS due to block size and metadata, while becoming increasingly slow to query and process. Swapping to SQLite as a backend solved the probem, all with a simple payload agnostic schema to organise blobs.

jc00ke

SQLite almost has this built in with it's "archive" feature: https://www.sqlite.org/sqlar.html Not the distributed part of course, but still, kinda cool that SQLite has this feature. I've always thought it would be cool to wire up Litestream to build a simple CDN.

vladislavp

"..SeaweedFS: This was the closest candidate. However, there were a few blockers. Our use case is overwrite-heavy (a user changing their workspace) over the same key, and SeaweedFS keeps appending before compaction, which can cause disk usage to grow uncontrollably. Blobs have their own IDs, which then have to be mapped against user keys in a separate DB, adding another layer. It doesn’t support TTL per blob. Also, reasonably complex setup with many parts. ..."

Would appreciate if someone who knows SeaweedFS could comment on the above My understanding that:

a) SeaweedFS supports Cloud backed SQLlite model ( https://sqlite.org/cloudsqlite/doc/trunk/www/index.wiki) where the SQLite file is stored in an SeaweedFS S3 compatible bucket (because of this feature https://github.com/seaweedfs/seaweedfs/pull/12243 )

b) I do not quite understand why the particular use case of user preferences is overwrite-heavy, if a they use a model where each user has their own SQLite database (unless it is not the case and all 20 milliion users use the same SQLite database? ... I am not clear)