Job queues are deceptively tricky
25 points by typesanitizer
25 points by typesanitizer
I might be missing something here because I didn't have time to read it in detail, but by skimming through it, I think the root of the problem is this is not a job for a job queue. The types of jobs to be done needs to be adjusted by the capacity available. So the available worker(s) should be pulling/deciding the best thing to do at the given time and not get the work pushed to them.
Can you give a concrete example of where you've used a job queue that is selective about the work it does based on how much capacity is available?
I've primarily seen them used in two ways:
If the former and latter both don't fall under "job queue" for you, then it sounds like you're operating with a different definition than the one I'm using in the post.
I also don't understand the parent's framing. A job queue is nothing more than a way to add asynchrony to a task. There are practically infinite use cases for that, from "fire-and-forget" sending of emails, to precomputing expensive logic without blocking requests.
I believe the idea is that if you have heterogeneous tasks/jobs in the same queue, then if you want to mostly efficiently schedule said jobs, you may want to know how many "units of work" a job should require. Heavier tasks might be worth three times as many as a simple job. I've personally never done it this way and just ran separate distinct queues per job type.
This is a task for cron. A queue is the wrong thing which is why it’s harder than it should be.
In particular systemd timers have options to control what happens when a timer fires but the job is still running from last time, what happens if the system misses a timer because it was shut down, etc
The author is mixing up and conflating many different things which are not job queues.
Conceptually, a job queue has no awareness of its consumers.
Batching, scheduled work / cron are separate concepts which mostly do not mix well with job queues.
90% of the job queues I have seen out there, were just glorified ways of removing heavy lifting work from HTTP request-response loops. Because most we developers are not familiar with how to write a service/daemon, they often rely on cron or cloud based scheduling solutions. Of course this results in longer than necessary waiting times, and possible congestion at the fixed hour these things run. And in the case of batching, often times dropped work.
Just let the workers continuously pick up tasks from the queue. If the queue keeps growing, just add more workers.
Is it tricky? or did you make it complicated?
Perhaps there are constraints that make using an existing system ok, but the whole time I read the article I was just thinking “roll your own!”
Really high performance job schedulers are complex, but if you’re kicking off multi-hour jobs that run once a week, you don’t have to worry much about the overhead of the queue/scheduler, and can implement it yourself. It’s honestly easier to write the logic of scheduling this job once per week than to try and shoehorn it into a system where jobs are more ephemeral/non-overlapping.
I get that there are lots of nuances here but from my experiences queues are really the main thing that works well in any distributed system.
Whether it's Celery, ZeroMQ, SQS, RabbitMQ, those have often been the foundations of decoupling and scalability in any architecture.
These are stored in object storage, and a new repo can be set up on a machine by downloading the reference repo, and fetching a delta of changes for the tip of the default branch.
The author should look up git bundles, which are ideal for this use case.
Never used a real job queue, but I find it often useful to change a frame of thinking when studying this sort of problem. Don't think about scheduling a fixed lump of work. Instead, think in terms of priorities --- the work is infinite, you just can't do it all, your goal is to do just the highest priority job first. You of course hope to be able to do everything under normal circumstances, but priorities allow for graceful degradation under overload, and often get the useful results faster under medium load.
In this case, thinking in terms of "time" when I want to schedule, and using interval and concurrency knobs, makes me feel like I've already lost? "Scheduling longer stuff on weekend" is just a special case of priority system ("prioritize long-running maintenance tasks when the expected future load is low"), and if I can't express the more general solution directly, that creates a long-term change risk. If existing knobs work today, and I build a fancy system on top of them, but then, tomorrow, realize that I do need something smarter, I might get path-dependently stuck.
If the concurrency limit is 1, then we have three choices:
- Prefer New: Stop J1 and start J2.
- Wait: Wait for J1 to finish and then start J2.
- Prefer Old: Auto-cancel J2 and let J1 continue running.
This looks very similar to a question about CI config I saw recently at work. I think what you actually want is to distinguish between "enqueued" and "started" jobs--both are "not finished" but cancelling a started job means you discard work, which means you can get starved if new work arrives continuously. In Buildkite terminology we want skipping but not canceling.
In my head it's like the main loop of a game:
while true:
start = now()
render_frame(start)
sleep_until_time(start + frame_duration)
If the loop can't keep up, then the sleep becomes a noop and you always handle the latest item.