SQLite for Everything
30 points by ggpsv
30 points by ggpsv
One transaction, ten thousand inserts, one fsync.
Whether or not this article was written using an LLM, this LLM-style writing (for a lack of a better name) of arguments (along with the usual "It's not X; it's Y") is exhausting to read, especially when it's repeated a bunch of times.
The punchy sentences are a giveaway. I agree it is exhausting.
Also, I have observed LLM-written prose (maybe claude-written specifically) tends to use the second person, like
It’s in your phone. It’s in your browser. It’s in your car. It’s in the plane you flew here on.
a lot, much more than typical 100% human-written text that is not a sales copy. That sales-y tone adds to it being exhausting.
Maybe I'm just too dense to notice the shift but that sounds exactly like dozens of posts I have read the last 20 years. Maybe not good posts, but I fail to see how any of this is new.
Scaling is the part where people expect the article to get quiet, so let’s not
Don't you just hate when articles go quiet on scaling?
"Quiet" had already been overused ~ 5 years ago, e.g. quiet quitting (doing your job) and quiet luxury (about as useful as "premium"). Now, the LinkedIn-LLM feedback loop keeps inserting it into so much text it generates. "Quiet" suggests, or is supposed to suggest, a deeper understanding or that something is more substantial than it actually is. I see it as a cheap writing trick. Everything is quiet, everyone is curious, everyone has quantified metrics without any explained methodology, everyone writes broetry. We are being surrounded by a staccato cadence of nonsense: short, punchy nothings everywhere!
Okay, this might have been tangential to your comment but I felt like venting.
I've previously used SQLite for everything, though currently I'm trying to use Fjall (KV store, like RocksDB). Obviously a lot less featureful than SQLite, but if I can build the features I need on top of it, I can fulfill my irrational desire to not have C dependencies in my Rust programs.
I've been curious about Fjall, mostly as an alternative to RocksDB, but also as a potential replacement for SQLite. I feel you on not wanting C dependencies in Rust, they can be a pain in the build process. How has your experience been with Fjall as a replacement for SQLite?
This might depend on your specific use case, but do you feel any overhead from using Fjall?
I've only briefly read up on it, but my impression was that Fjall does more things in the background: in-memory caching, background tasks/threads, etc. Whereas SQLite seems like it only does "extra" stuff when you're actively calling into it. But again, that's just my impression.
It is recommended to configure the block cache capacity to be ~20-25% of the available memory - or more if the data set fits fully into memory.
That made it seem quite a bit heavier than SQLite. Though to be fair, I don't know how much SQLite soaks up in comparison!
I haven't noticed any overhead, but I also haven't tried throwing it at anything big. My sandbox for testing abstractions over Fjall is my Discord bot, which doesn't have very many users :)
Note that SQLite also does in-memory caching. The default is pretty small, though, only 2000 KiB. I haven't done any testing to see how much cache SQLite and Fjall require to get good perf in different workloads.
As for background tasks, my understanding is that LSM trees require maintenance, while B-trees don't. A B-tree might end up with subpar performance without maintenance, but it'll basically be Fine. Since LSM trees require it, Fjall runs maintenance tasks in one or more background threads. If it didn't, the maintenance would need to be done in a blocking manner while you're trying to do an insert or something, which would randomly add latency.
desire to not have C dependencies in my Rust programs
Are you adventurous enough? https://github.com/tursodatabase/turso
I know multiple people who use Turso in production and say it's super stable and all, but I'm still a little spooked out honestly.
PostgreSQL for everything. It’s easy enough to start in-process.
That's not what people mean by in-process typically. This is just running the full database as a child process.
This is in-process https://pglite.dev/
I currently use SQLite for everything as well. It comes bundled with Bun and dealing with it is a breeze. It’s just a file, I can rsync the production db backup from the server, every backup is the sqlite itself. I love the simplicity of it.
Latest Node.js releases included a dedicated module for running SQLite "inside" Node.js without external dependencies. See https://nodejs.org/docs/latest/api/sqlite.html