Let Me Let You Own Your Data
21 points by thomas0
21 points by thomas0
Most people, if required to determine the hosting for their own data, would prefer to hand that responsibility off to another party that specializes in things like availability, redundancy, resilience, etc. Even self-hosters tend to entrust their remote backups to a commercial party like Backblaze.
Besides, all of these options compromise on the data model anyway, being KV or blob storage when all I really want is an API to run SQL queries
Very silly option, but there's prior art for running SQLite on top of blob storage like S3 by presenting a VFS to SQLite that interfaces with the blob storage backend.
Most people, if required to determine the hosting for their own data, would prefer to hand that responsibility off to another party that specializes in things like availability, redundancy, resilience, etc. Even self-hosters tend to entrust their remote backups to a commercial party like Backblaze.
I tend to agree with you, I self host a couple of things but it's a constant stream of work (even though it might be as simple as running apt update && apt upgrade.
Very silly option, but there's prior art for running SQLite on top of blob storage like S3 by presenting a VFS to SQLite that interfaces with the blob storage backend.
This led me to lookup what you were talking about and found that SQLite supports Azure and Google blob storage (but no AWS?), https://sqlite.org/cloudsqlite/doc/trunk/www/index.wiki
SQLite constantly surprises me with what it can support!
Offhand I know that Azure blob storage happens to include an API for block IO (i.e. like a SAN or like EBS) and some other primitives you could use for a VFS but I can't remember what they all are
but S3 only does immutabl-ish object storage, by which I mean you can overwrite an object but iirc you have to overwrite the entire thing; last I checked you can't just overwrite part of an object in S3
I was working on something related as a hobby project. My thinking was exactly this:
What I got working was a "personal data server" (a bit like Solid Project's), but the API was generally just SQL, instead of a semantic Web layer. This seemed most practical. Under the hood, it was a Go server that spawned HTTP adapters to hold open an SQLite database on behalf of the user's app. It would close the DB when not needed to conserve resources, and was built as multi-tenant from the ground up, so each user/app pair would get their requests routed to their respective "shard".
It never really got anywhere, because by the time I got the implementation underway, two things dawned on me:
I haven't really dabbled in this for some months, but it is an area I'm likely to revisit some day.
I built a proof-of-concept social network where data was stored locally on IndexedDB, but it was also kept in encrypted form by any friends you had on the network. If you lost your laptop you were able to login with the public key (the sign up page generated a PDF and would tell the user to confirm having printed it), and retrieve your data from your friends whenever they were back online (using WebRTC).
I think this approach solves the problem where people don't want to host their own data because they could lose it -- or is at least a good compromise, since all the user needs to save is the public key.
Not advocating for it (never used it) but a user being able to "have" their own data is party of Tim Berners-Lee's Solid project.
https://solidproject.org/about
I've always meant to spend some time looking into it.
I feel like there's a contradiction in requirements here. Either the user lets us hold their data (i.e. we throw it all in a DB), or they hold it on their device (and lose cross device sync), or they tell you how to get it (and log in/provide keys for some other provider).
I bet it could be done with cryptography somehow, but that has its own issues.
I've been banging my head against this problem for the past year as well, trying to make my app store and sync data through whatever file syncing service the user (probably) already uses (Dropbox, Google Drive, iCloud, etc).
Alas, I didn't really manage to make it work. One challenge is that file syncing is quite unpredictable (when files sync, in which order they sync, etc), and another is that it's not easily accessible on mobile, which is why I want syncing between devices in the first place.
The approach I now landed on is syncing through any S3-compatible service, which is not really accessible to "normal" end users (not my target audience anyway), but at least it's more or less a standard with lots of implementations, and other than creating a bucket the user then doesn't have to host anything.
There was WebSQL Database, but Mozilla and Microsoft conspired to kill it and replace it with the far, far inferior IndexedDB back when they still had some relevance to web standards. But in any case an in-browser database is not something durable enough (e.g. proper backups) to rely on for users' critical data.
My understanding is that WebSQL was killed because the spec was just "whatever SQLite does", so it was hard to have it be a web standard.
That could have been addressed by documenting what it is SQLite does. Going from WebSQL to IndexedDB is like going from SQLite to gdbm or BerkeleyDB, except IndexedDB itself is implemented on top of SQLite so you get all the overhead and none of the expressiveness.
One option that's worth considering here is GitHub. Private repos are free, and you can OAuth with a user's GitHub account and store data directly in their own repo.
The downside is that most non-technical users don't know what "GitHub" is and sending them through that signup is weird.
If your users are likely to have GitHub accounts already it could make an excellent backend for storing data.
An API to run SQL queries isn't compatible with people hosting their own data. If people host their own data any time soon it's going to be in simple JSON blobs with extremely basic formats that are shared across consumers of this new type of thing. It's a bit like SQL/NoSQL. If you want the speed advantage of NoSQL, you lose the power of being able to run arbitrary SQL queries. If you want complete arbitrary localisation and distribution of data, you're not going to be able to query it, search or index in it in any way that's recognisable to web apps today.
Couldn't you store the user's data as an encrypted blob in some public place? Queries like sqlite/duckdb can run locally on the browser.
Cross device sync is a QR or a way to share the private key (+password optinally?)