File d'attente - file-based job queue tool

35 points by mccd

Relax

Finally, a job queue where jobs are not defined by decorated python functions.

zimpenfish

I just archived our repo with that in and I pray I never have to deal with it ever again because it has beknighted my life for going on a year now and I would rather eat my own eyes than go back.

(assuming you're talking about C....y, of course; I shudder to think there's more variants of its horror available)
- Relax
  
  Not just celery, there are a lot of them. Coupling everything to python is such an odd choice, and not only that but decorators are kind of awkward in a lot of places. For example, all of the decorator-based python web frameworks do weird things to deal with the circular dependencies that pattern creates.
dgl

Notification on failure. Unfortunately inotify does not work with fuse, which would have been elegant otherwise

I think this is a limitation of bazil/fuse -- https://github.com/hanwen/go-fuse does support inotify, it has Notify... operations on the inode anyway, I can't say I've tested it in detail but it should be possible.
- mccd
  
  Interesting, thanks for sharing. I noticed go fuse also has freebsd support which would also be nice.
krig

How does the job runner know if a file is fully written before it tries to run it?
- mccd
  
  The file lives in memory until the open file handle is closed, then it gets written to the db.
nickmonad

Neat idea!

If I'm inspecting an active job with a text editor, and the job completes, but then I save the open active job file, does it run again? Or is there a check in the DB to see if that job has been completed already?
- mccd
  
  Hey, it should lead to a permission denied error, since only filed can write to the active directory.
  
  I haven't added it yet (actually should be in the to do) but the intention is to only support remove operations for the active directory, which could be used to kill active jobs.
denz

Very cool! I'd be curious to know some real world use case for this!
- mccd
  
  My hope is that's its useful in situations where you're building a webapp in a programming language without a decent job queue library available.
  
  I also think the admin part feels very ergonomic. Files are a bit easier to manipulate with grep etc. So the system is easier to extend, and you don't need to log in through some admin portal in the browser to inspect or rerun jobs (which means you also don't need to manage the auth around it).
sigmonsez

no mention of concurrency, is it single worker or can there be multiple jobs run in parallel ?
- zk
  
  Taking a very very quick, cursory look at the code it seems it does/can run multiple jobs in parallel.
- mccd
  
  Yes, multiple jobs in parallel are supported, and you can tweak the amount of concurrent jobs as well in the config.json
  
  https://git.sr.ht/~marcc/filed/tree/main/item/filed.1.scd#L80
  
  Sorry I should mention it a bit clearer in the readme.