libeatmydata - disable fsync and SAVE
23 points by bakaq
23 points by bakaq
SQLite has a command to enable this — iirc it's pragma synchronous(off)
Use this in various disk-io heavy tasks where speed is more important than integrity... saves some time in various batch-jobs. (And a 'sync' wil run afterwards to make sure everything ends up on permanent storage)
I used to use this with Notmuch before I had an SSD and the Xapian db was on sinning disk. It made a big difference in performance and wih regular backups the risks were well understood and acceptable.
Not sure if this needs the satire flag, because it looks like something that could be useful in some circumstances. I can't think of any though.
Really handy for CI testing jobs
That's true, but this should be handled by the CI setup by making fsync a noop so it's optimal for everyone by default. Either make the CI jobs write only to a tmpfs, or write to a filesystem mounted with nobarrier (assuming I understand what nobarrier does; documentation is scarse)
And this is a mechanism for the CI to make it a no-op. One that requires no system administrator permissions, and can therefore be done by any part of the pipeline.
But it is the system administrator's job to do it, so they do it once for everyone. If eatmydata in a CI, it means the CI's administrator did not do their job. Like, if a CI's logs is broken, so the best solution is to make the admin fix it, not make every repository's developers to configure an alternative logging system.
And eatmydata does not work for statically compiled executables (eg. Go), so in that case you need to either patch your executables or ask the CI's administrator to disable fsync.
System administration hasn't worked that way in at least 10 years; the entire point of the current devops systems is to decouple system administration from decisions on what runs where and how. You can debate whether this is a good thing or not, but it's the world that we have today.
There is no "optimal for everyone". I've written tests for concurrency issues where fsync is required to pass. This can be only used by people choosing which app is running and in what way. It can never safely be a default.
I use ZFS with a dataset with the nosync option for builds for a similar reason and do builds in the directory where it’s mounted. This also has the nice property that it’s trivial to exclude from backups. If power goes out or the machine crashes and I lose a load of .o files, I don’t care even slightly. In the absolute worst case, I can delete everything in that directory and do clean builds.
It significantly speeds up debootstrap (a tool to build a Debian system) on HDDs. Debootstrap calls APT which does a lot of small writes and fsync sequentially because it's usually important; but I don't care about corrupting an image if I lose power while building it.
The performance gain is lesser in the SSD era, but still exists.
Batch jobs where you only care about the final results. In the unlikely case the node crashes, just start over.
Analytics jobs where the source of truth is somewhere else, but you're generating scratch data for an analysis of some sort.
Useful for caches that you use across reboots -- at least, that's how I've seen it used. Not satire, this is distributed widely.
Another one I've run into was restoring an SVN repo from a backed up dump file. svnadmin load reads a dump file and replays all the commits in it to create a new copy of the repo from the dump. This is really slow on HDDs because it does a full commit-to-disk with fsync()ing after every commit, but you don't need that for restoring from a backup - if the computer crashes during the restore you'd just delete the half-restored repo and start over. So you can make it go much faster by LD_PRELOADing libeatmydata to stop it spending time in lots of fsync() calls, and instead just run the sync command once at the end to flush everything to disk.