You Don't Love systemd Timers Enough
108 points by edwardloveall
108 points by edwardloveall
Note that the author uses NixOS as hinted at in many places and I think this is one of the main reasons why they like systemd (timers) so much. NixOS makes it very easy to declare what systemd setup you want in a single file with all the flexibility of a programming language and NixOS makes sure the right systemd unit files show up at the right place and get removed when you don't declare them anymore.
You can see an example in the NixOS wiki. Those snippets all go in your system configuration and if you make mistakes you even get validation errors while building (well ahead of deploying). You also don't need to worry about PATH because you'll typically refer to packages inside nixpkgs (or some other source of nix packages), which means they will be spliced in with their absolute path in the nix store and again, you'll get a build time error if you make a mistake.
IMO NixOS completely removes all the major hurdles of setting up any nontrivial machinery with systemd.
At this point the systemd haters peer out of the woodwork in anticipation of torpedoing timers because they are part of the systemd project and because they replace mature (if clunky) technology. I'd rather not spend our time arguing about cron, so briefly consider why newer solutions like systemd timers that benefit from years of hindsight are better:
I feel like this is the case with many parts of systemd. It by no means is perfect, but lots of its design decisions are based on learnings from the (more traditional) past. I recently re-listened to CRE’s (a German podcast) 2015 episode with Lennart Poettering, where he explains the reasoning behind some of it, and can still recommend it today.
Most of systemd's problems are arguably UX problems. The sad part is that systemd is actually very capable, like half the things people use docker (or even kubernetes) for systemd does objectively better, but it just chafes so much when you try to do anything with it.
As it's something that the median user will only interact with infrequently, something like systemd needs to be extremely intuitive, as most people outside of maybe distro maintainers will never get the repetition needed to learn everything by heart, they're only ever piecing stuff together on the fly from the documentation, and that documentation isn't great (and what's documented is very sprawling and difficult to get an overview of), and systemd isn't intuitive.
So because of this, absolutely everything with systemd chafes for the average user. You can't just read the code to see how it works, you're always looking up magical conventions and strange files in unusual file formats scattered over the etc directory, and it's not made better by the fact that there's hidden state that you need to know the correct invocations to interrogate, and nothing of this can be done with the regular tools and utilities of the system (which people might know), but requires a set of custom tools that almost nobody ever properly learns and that don't behave like anything else on the system..
and that documentation isn't great
In my experience I'd be hard-pressed to find a single undocumented systemd feature / setting. It's incredibly thorough and comparatively high quality. Even obscure edge cases when I was convinced I found a bug were usually documented.
I will admit it is sprawling.
It's certainly exhaustive, but that's arguably part of the problem, particularly because it's so exhaustive (and sprawling), finding the appropriate man page can be quite the task. As your sibling comment notes, there's nearly 300 manpages that could be the one you're looking for (and many of the man pages are exceptionally long on top of that).
I love systemd's man pages, they are undeniably awesome. For those who aren't confident in their ability to just use the man pages to do what they want, systemd's website also has some additional non man page resources. This includes a linear series of posts called "The systemd for Administrators Blog Series".
I agree with most of what you wrote. The documentation however really is great! Every command, file etc is extensively documented:
~> man systemd<TAB>
systemd (1: systemd system and service manager)
systemd.automount (5: Automount unit configuration)
systemd.device (5: Device unit configuration)
systemd.directives (7: Index of configuration directives)
systemd.dnssd (5: DNS-SD configuration)
systemd.dns-delegate (5: DNS Server Delegation Configuration)
systemd.environment-generator (7: systemd environment file generators)
systemd.exec (5: Execution environment configuration)
systemd.generator (7: systemd unit generators)
systemd.image-filter (7: Disk Image Dissection Filter)
systemd.image-policy (7: Disk Image Dissection Policy)
systemd.index (7: List all manpages from the systemd project)
systemd.journal-fields (7: Special journal fields)
systemd.kill (5: Process killing procedure configuration)
systemd.link (5: Network device configuration)
systemd.mount (5: Mount unit configuration)
systemd.mstack (7: Mount stacks in self descriptive directories)
systemd.negative (5: DNSSEC trust anchor configuration files)
systemd.netdev (5: Virtual Network Device configuration)
systemd.network (5: Network configuration)
systemd.net-naming-scheme (7: Network device naming schemes)
systemd.nspawn (5: Container settings)
systemd.offline-updates (7: Implementation of offline updates in systemd)
systemd.path (5: Path unit configuration)
systemd.pcrlock (5: PCR measurement prediction files)
systemd.pcrlock.d (5: PCR measurement prediction files)
systemd.positive (5: DNSSEC trust anchor configuration files)
systemd.preset (5: Service enablement presets)
systemd.resource-control (5: Resource control unit settings)
systemd.scope (5: Scope unit configuration)
systemd.service (5: Service unit configuration)
**…and 263 more rows**
It's just that no one uses man pages anymore…
I think this is a generational issue. Back when I first got a Raspberry Pi, I checked some forums for solutions to problems, and it seemed to me that the vast majority of users where people whose first time exposure to a Unixlike was via Raspbian, which uses systemd. So there was not a lot of people remembering init systems from before.
I've recently standardized my home machines on DietPi which similarly doesn't have any legacy init background, and using systemctl and journalctl is slowly becoming more intuitive for me personally.
I just wish systemd timers had a lower initial overhead, especially when it comes to user management. It's really hard to beat crontab -e, when looking at how much setup systemd timers need.
Systemd's multiple config files and services to set up a timer are... an insane API choice.
We clearly have different ideas of what "insane" means.
Separating the minutiae of when something runs from how something runs gives a lot of flexibility, particularly when combined with things like templated services, and override files.
Sure, but they could provide a concise way when I don't need the flexibility and just want to run a command; like systemd-fstab-generator which generates .mount units from a single line in a file.
There's a "service" which determines what runs. There's a "timer" which determines when to trigger a service.
It's useful to separate these two so that you can have multiple timers attached to a single service.
Maybe it would be nice if you could create both in a single file for the common case of a 1:1 mapping. But having roulette.service and roulette.timer in the same directory is fine.
It also means that you can systemctl start roulette if you want to manually run the service, which is useful for debugging and also if it's something like backups that you may want to be able to manually run.
I hate to be that guy, but this is a lot less annoying when using NixOS - all in one place (easy to copy&paste), slightly less verbose by default, you can create trivial helper functions if needed.
stdout and stderr output often ends up in a black hole (and, often, sent to the host's mail system, which is usually not what you want to happen.)
Call me old-fashioned, but I still set up email on servers to reach me. Once you have it automated (so every new host gets it for free), it's handy regularly.
(E.g. like open a multiplexing thing, run long_running_process | mail root@localhost -s "done $?" and forget about it...)
I spent a long time worrying about how to systematically get logs from my cron scripts until I realized… just use systemd timers. Logging solved. I see no reason to ever use cron again. I wish I had found out earlier.
Pipe to logger or >> to logfile or leave the defaults and get emails?
Former doesn’t address stderr, latter is not my definition of logging (and I’m otherwise a proponent of system mail)
I'm a dyed in the wool systemd-unenjoyer, but I'll give that systemd.timers are "one of the better" concepts of the product, so when the author covered themselves in enlightenment by deriding those with valid grievances I was taken a little aback.
I do like using it in conjunction with the at command though, at for one-off commands executed at a specific time, systemd timers and simple unit files for everything else.
The only major improvement I'd love to see is which user has a timer running. Though granted I am one of the very few people who runs a shellbox in 2026, it would still be instructive to understand which user created a timer that hammers the disk every second.
The only major improvement I'd love to see is which user has a timer running.
Could you use user units for this purpose rather than everyone installing system timers? I think with loginctl enable-linger those can run without an active user session. (Though there are certainly use cases where that wouldn't be sufficient, and I don't know yours.)
Yes, exactly, the user timers are what I mean!
Listing all the timers is one thing, but knowing who owns them is tricky.
I'm surprised there has been no mention of Shepherd timers. They solve all of the mentioned problems, in an arguably more elegant and flexible way.
That's the first time I came across GNU shepherd – is there any way to mix and match or would one need to replace their init system with shepherd?
It's no secret that I have a lot of criticism of systemd, but I can also appreciate the bits that don't deserve criticism.
And certainly for trying to improve over cron, systemd timers don't deserve criticism.
But for being just as confusing (in new and innovative ways) and about as difficult (maybe even more?) to get right if you almost never reach for them, they definitely deserve criticism.
The only thing they have going for them is truly that they have more features and more modern features.
It obviously doesn't fit the systemd operating model, but snooze is the closest I've found to a periodic timer thing which doesn't require me to reach for, and carefully study, a man page (the Usage: output is enough for 99% of tasks) to avoid footguns.
And, if you are using a daemontools derived service manager, it's just one file rather than two (although, again, I understand why there are two in systemd).
Nice post, I have a draft of a very similar post that I had to reference recently.
One tip if you go down the systems rabbit hole like I did, is to syslink your units files and timers from a folder associated with the relevant project into /etc/systemd/system/. The one complaint I have with systemd is that there’s no distinction between distribution-installed units and manually-written units. But you can maintain that separation yourself with syslinks.
Actually, the path is the distinction. System/package/distro units go into /usr/lib/systemd/system, local overrides or local units go into /etc/systemd/system.
Thanks, that sounds kind of familiar. There's a lot of junk in /etc/systemd/system that I don't recognize but a lot of it is symlinks into /usr/lib/systemd/system/ which I guess don't count?
Some of the junk might also be results of running commands like systemctl enable/mask/edit etc. for distro units
Speaking of, is there a particular reason why it’s /usr/lib/systemd/system and not /usr/share/systemd/system? I would more commonly expect system things that are overridden in /etc to be /usr/share, unless they contain architecture-specific files, which systemd unit files aren’t.
Another benefit is that if you have a task that must only run one instance at a time, that's the default behaviour with systemd timers. This is great if somehow your task hangs; you won't come back to find that cron has still started a bunch of new instances.
One issue I found is that while systemctl list-timers is great, it seems to take a while to update sometimes, which makes the debugging loop painful. Maybe that was just a random issue in my distro that's fixed by now.
stdout and stderr output often ends up in a black hole (and, often, sent to the host's mail system, which is usually not what you want to happen.)
It doesn't seem that the author knows what he is writing about.