jobber - execute and manage a list of tasks as shell commands
jobber [options] <jobfile> [actions]This works on a job consisting of a number of tasks in a managed queue with filesystem-based synchronization for parallel instances possibly on multiple compute nodes. A task is simply a one-line shell command in the job text file. Lines commented via leading # are ignored. Continuation lines are not supported. The tasklist file name is mandatory, followed by one or more actions, which are executed in order if nothing fails:
initialize, create fresh queue and control files
run all tasks in queue (same as -1)
run this many tasks in this instance, -1 means all
execute specified tasks if still queued
stop operation (do not start new tasks)
start operation (after stop)
exit with zero (true) if there is more to do right now
print number of matching tasks (default: all)
print number of tasks left in queue (count:queued)
print number of tasks being worked on (count:running)
print number of finished tasks (count:done)
print number of failed tasks (count:failed)
print number of skipped tasks (count:skipped)
print number of held tasks (count:held)
print grouped runtime stats
print a tabular report
put tasks back into queue (defaut: all)
requeue done tasks (requeue:done)
requeue failed tasks (requeue:failed,skipped)
requeue tasks marked as running (requeue:running)
(Make sure you do not requeue tasks with really active workers!)
hold specified queued tasks (default: all)
release specified held tasks (default: all)
append task lines from standard input to the job
look for added tasks in job file
try to restore queue file after worker death
remove queue and control files
print tasks as shell script (default: all)
compute CRC values for lines on stdin (job file ignored)
figure out and print next task
If the job file does not exist, the init action will create an empty one that you can fill using the append action.
A <spec> is a comma-separated list or the word 'all' to include all tasks, one of 'done', 'failed', 'held', 'queued', 'running', and 'skipped' to select tasks of that state, a numeric task ID or a range separated by a hyphen (a-b, or even a- for starting at task a and going to the end), or more explicitly <field>=<value> for fields state, id, ret, exit, signal, and crc. States may be appreviated with the first letter only. Multiple specifications select tasks that match any of them. The values for ret/exit/signal are only checked for tasks in failed or done states.
As long as a job is being worked on, the existing contents of the job file are expected not to change. Appending is OK, but best done explicitly via the synchronized jobber append operation. If you change existing commands, possibly moving offsets in the file while at that, chances are that you will confuse or at least irritate the workers and yourself later The workers will skip tasks with changed command lines (judged by CRC).
The refresh action is supposed to add tasks that have been appended to the job file after queue initialization. Existing commands are not checked for being changed. The workers do that before trying to execute and it would complicate state semantics if we interfered.
The exec action prepares a fixed list of tasks first and then starts executing them, ignoring further additions of egilible tasks at runtime. There lies the difference between all and exec:all (or just all and exec).
The print action without further arguments produces a shell script with all commands, but only those for queued tasks uncommented. Also, a header with some global stats is included int that case. A limited selection of tasks yields a shorter printout with all of them uncommented. Return values are parsed and actual exit status or kill signal presented for finished tasks.
The report table contains task and worker ID, runtime, start time, CRC and exit status or signal. A value of -1 is put in where there is no sensible value. Tasks running right now show a start time, but no runtime yet.
You can temporarily exempt tasks from execution (putting on hold)via the hold command. Releasing them restores the queued (pending) state.
The main process creates a child process for each task, possibly in parallel. Signals INT, HUP, and TERM in the main process trigger TERM to all children to end things gracefully, with the abortion of tasks redcorded in the queue file. Stdout and stderr output is serialized by the parent to produce intact lines. Text output is assumed. Binary data should be redirected in the task command lines. Standard input is passed through only for non-parallel work. =head1 PARAMETERS
These are the general rules for specifying parameters to this program:
jobber -s -xyz -s=value --long --long=value [--] [files/stuff]You mention the options to change parameters in any order or even multiple times. They are processed in the oder given, later operations overriding/extending earlier settings. Using the separator "--" stops option parsing An only mentioned short/long name (no "=value") means setting to 1, which is true in the logical sense. Also, prepending + instead of the usual - negates this, setting the value to 0 (false). Specifying "-s" and "--long" is the same as "-s=1" and "--long=1", while "+s" and "++long" is the sames as "-s=0" and "--long=0".
There are also different operators than just "=" available, notably ".=", "+=", "-=", "*=" and "/=" for concatenation / appending array/hash elements and scalar arithmetic operations on the value. Arrays are appended to via "array.=element", hash elements are set via "hash.=name=value". You can also set more array/hash elements by specifying a separator after the long parameter line like this for comma separation:
--array/,/=1,2,3 --hash/,/=name=val,name2=val2The available parameters are these, default values (in Perl-compatible syntax) at the time of generating this document following the long/short names:
[]Which configfile(s) to use (overriding automatic search in likely paths); special: just -I or --config causes printing a current config file to STDOUT
''time limit as a given point in time, overriding --time, value being a date specification that is just passed to date -d (use @123 to specify epoch 123, preferrably ISO date format otherwise)
2End operation after too many execution attemps (i.e. bad shell, fork issues), number multiplied with current total task count, zero disables the limit for possibly endless looping on automatically requeued tasks.
0fail and stop execution on the first unsuccessful task
0Do things as requested even if there are signs against that.
0Show the help message. Value 1..9: help level, par: help for paramter par (long name) only.
Additional fun with negative values, optionally followed by comma-separated list of parameter names: -1: list par names, -2: list one line per name, -3: -2 without builtins, -10: dump values (Perl style), -11: dump values (lines), -100: print POD.
0show execution history for task print and report (the latter dropping tasks not executed yet)
8192Block size for reading messages from worker processes.
''Put stdout/stderr of tasks into the specified directory (if non-empty), with file names <task>.<worker>.out and <task>.<worker>.err, read stdin from <task>.in (or /dev/null).
0Label task stdout/stderr line printouts with the task ID, also the worker ID for value > 1, or with string matched from the command for value > 2 and non-empty labelexp.
''Perl code to evaluate inside sub { } on command string as first argument, scalar return value being used as label. This is used instead of labelexp with label > 2 if non-empty. You got all the power of Perl. Be careful.
'\\bssh\\s+([^-][^=\\s]+)'Regular expression to match against command string to extract a label, the first capture group ($1) is used with label value > 2. The default tries to capture simplest SSH usage and show the host name.
0abort lock attempt after that many seconds, ending total operation
60warn about locking taking over given amount of seconds
''prefix to print before jobber messages to distinguish them from your command output
0Disable line-buffering of task standard and error output. This may insert line breaks to show the partial lines earlier (when not passing through the descriptors in serial mode without labelling).
0no standard intput/output/error at all to/from tasks (/dev/null)
1Start that many processes for working on tasks in parallel, just like repeating the jobber command for multiple background tasks. All processes work on the given count of actions together. A negative value means to start as many processes as tasks being planned at startup.
0Pause for that many integer milliseconds before forking a worker (with locked queue).
0no non-error messages about task execution (overrides verbosity)
''Expicitly set the shell to use for the task commands. This is fixed in the queue file on init. It needs to accept -c "command line" but nothing else is required regarding its syntax. If none is specified here, a #! line in the job script will set the value, falling back to the SHELL environment variable and as last resort /bin/sh. That value is split once after the first whitespace to separate interpreter path and argument, just like it is customary for script parsing.
0reflect unsuccessful tasks as overall non-zero exit value
1Use file flush and sync to safeguard changes to the job queue. Disable at your own peril.
'none'time limit in seconds; stop running tasks if given time is running out ("none" disables time limit)
'2max'how to estimate the expected runtime of a task, counting towards the limit, either a fixed number of seconds or a factor followed by one of min, mean, and max (like the default 2max) for using the minimal, mean, or maximal past task runtime as base
1prefix messages with a timestamp
0When printing all tasks, do not comment out command lines of tasks that have been executed.
0increase/set verbosity level (up to 3)
0print out the program version
The motivation for this tool was the need to work on a predetermined set of independent computations with a scientific simulation program covering parts of some parameter space. Things should be documented and reproducable. Parts that fail should be fixable and then re-startable without re-running everything. There might be differing instances of worker processes on multiple compute servers picking tasks at the same time. One approach to such a problem are array jobs in batch systems, where you need to decide on a mapping of array ID to parameter set when setting up the job. Pre-generating the program configurations as shell command lines decouples the decision what has to be computed from how it will be distributed on some cluster and put into action. It also helps manual testing and reproduction outside the batch environment.
Using jobber, you have a simple sub-scheduler that can make better use of your batch system time slots, pushing through a number of your individual computations of uncertain (sometimes very short, sometimes longer) duration where otherwise an array job might have to be rescheduled for each small one. The clear recording of the success of each job task avoids you having to sift through log files and matching up batch system job IDs to the actual tasks computed. There is some othogonality to exploit, having the batch system distribute your work, deciding when and where it runs, and having jobber decide and record what gets actually computed. When you have some thousands of tasks to get through, with expected failures and restarts along the way, this can be rather helpful.
Outside the realm of batch systems, there are other tools that mainly help you running a number of tasks in parallel in an ad-hoc fashion. If you have a common command and a set of arguments to distribute to instances of that command, and just want to get through them without much fuss you might be happy using xargs or GNU/moreutils parallel. A strength of those tools is that they can construct the command lines to run on the fly. This is in contrast to a situation where you might spend days figuring out the correct commands and arguments to use, iterating over details with test runs and refining the final set of tasks.
A Make-like tool is a better solution when your tasks have interdependencies and the correct execution order along with possible parallelism need to be figured out from your description of sources and targets. This does not have to relate to building software. Any kind of file can be a Make target. A Makefile also provides documentation and reproducibility, like a flat shell script serving as input to jobber.
Jobber can be used to directly run things in parallel in one instance, but also many jobber instances can be started independently on differing computers to work together on the job queue on a shared POSIX filesystem. The latter needs to provide atomic rename and fsync, which is a bit more common than proper distributed locking (which you might have neutered with NFS mount options). The point is that you run the same command everywhere, e.g.
jobber /path/to/jobfile allto run the next tasks from the queue, arbitrarily prepared beforehand. Jobber does not distribute the work among networked machines itself. If you want work to happen on a remote box, you either arrange for the jobber instances being started where desired or just put ssh into the command lines to run one instance as a central dispatcher. You got the power of the shell to play any tricks there, jobber does not care.
The file containing job tasks is supposed to be a simple shell script. It may start with a shebang line to indicate the shell to use. The shell must accept a command via the -c parameter, as in
/bin/sh -c 'echo hello'Further lines starting with # (possibly with leading space) are ignored as comments. All other lines are taken as commands to be executed. Line continuation is not supported. So,
echo hello
worldwill be interpreted as two independent commands, the first one ending with . A prototypical task list file for your job looks like
#!/bin/sh
# This job is important. This comment not.
echo "step 1"
for s in $(seq 2 5); do echo "step $s"; done
echo "step 6"
# This is the end.Jobber creates and works on a queue file that contains pointers to the command lines in the job file. You are responsible for not changing the latter in ways that confuse operation. As a safeguard to spot modifications that would result in corrupted command lines, checksums of the commands are verified before executing them. If the checksum does not match, the task is marked as skipped instead.
The file format is designed to be somewhat readable as plain text but also to enable recording of task execution status in-place without having to move other file contents around. This means that header fields are padded to expect possible values (20 decimal digits for 64 bits), as is the return/exit code field for a task. Choices have been made to avoid unnecessarily long lines for each task entry. An adequate CRC16 value for modification checks instead of some long cryptographically secure hash is one such choice.
An example with a few tasks:
JBQv2 /bin/bash
cnt 00000000000000000003
twk 00000000000000000001
tdn 00000000000000000001
tfl 00000000000000000001
tsk 00000000000000000000
nwk 00000000000000000002
mxt 00000000000000000452
mnt 00000000000000000321
smc 00000000000000000002
smt 00000000000000000773
nxt 00000000000000000266
Q 1 130 12804 00000
W 2 161 34222 00000
D 3 234 00783 00000
F 4 313 12364 00001
B 2 1633875385 2
B 3 1633875386 3
E 3 1633875796 3 0
B 4 1633875796 4
E 4 1633876158 4 1All fields are separated with tabs to ease ad-hoc parsing using grep and cut. As plain text, the file format is portable between different types of machines, as long as their execution environment agrees on the line ending and alphanumerical encoding.
It begins with a file type identifier and the shell the command lines are supposed to be executed with, usually taken from the environment at queue creation time or, if the first line in the job file looked like it, from there. Directly after that the queue statistics lines follow, with these numerical fields, each on one line:
Total count of job tasks.
Job tasks being worked on (marked as running).
Successfully finished job tasks.
Failed job tasks.
Skipped job tasks, be it because of some inconsistency (CRC failure) or a possibly temporary issue forking/launching the shell.
The number of workers workers for this queue. Each one increments that value and then has a personal ID to write into lines. Workers ending operations do _not_ decrement the value, so that it stays unique. There is one worker started per task, so that the combination of task ID and worker ID (reported as task.worker in diagnostic messages) is unique for the whole job run up to the next init.
The worker ID is basically a counter for task runs, be it different tasks or the same task repeatedly.
Maximum recorded task execution time in seconds.
Minimum recorded task execution time in seconds.
Job task count that went into the execution time sum. This includes all task runs that went into accumulating the overall execution time, possibly inflated by requeueing tasks. The purpose is to get a meaningful mean execution time when dividing the sum.
Sum of all execution times in seconds.
Whole-file byte offset of next entry to work on. To avoid reading through the whole file all the time, the next entry's offset is stored. Each operation that modifies task state needs to update this. A value of zero indicates that there is no known next task and a worker should indeed parse through the whole file to find it. An obviously invalid value is treated the same way. The search continues from the indicated position on and then re-starts from the beginning if the value was non-zero.
The times are initialized with zero. If they are still zero when the count of done or failed tasks is non-zero, this indeed means that the tasks finished in under one second.
After this header the task state records and the execution log lines follow. Each command line in the job file translates to one task ID (job file line counter, disregarding comment lines). Each of the state and log lines starts with an uppercase letter followed by a set of numbers. The first number always is the task ID.
The task state records are led by the letters Q, W, D, F, or S with these meanings:
The task is in the queue, ready for execution.
It is in work currently.
The work is done, the task finished with good outcome.
The task execution failed (details in the return value field).
The task was skipped. The only proper reason for that right now is failure to verify the CRC of the command line (changed job file).
The task ist on hold. It can be scheduled for execution by releasing it into the queue again.
A full state record is comprised of the fields
<state> <id> <cmd> <crc> <ret>with these meanings:
The state code, see above. It is modified in-place when execution state changes.
Job file offset to locate command line.
A CRC-16 value of the command line. Execution is not attempted if that does not match. This field is of fixed width 5.
The 16-bit return value following the waitpid() format ($? in Perl), also fixed to width 5. It starts out as all zeroes and is updated in-place after execution.
The log entries begin with uppercase letters with these meanings:
Execution of task began.
Execution of task ended. The 16-bit return value of this attempt is also appended to the line.
The task is being rejected by the worker and marked as skipped.
The task execution was aborted without doing any work, the task having been requeued. This happens when the actual shell execution did not work.
Each log entry is completed by a timestamp (epoch seconds) and worker number, and the 16-bit return value from this attempt.
The lines
B 3 1633875386 2
E 3 1633875796 2 0from the example above inform you about task 3 being started by worker 2 at 2021-10-10T14:16:26 UTC and execution ending at 2021-10-10T14:23:16.
These log entries are appended as operations go on without disturbing the task state records that are by construction preceeding the log entries. Note that state records and log lines can still be intermixed as new tasks are added to the queue.
By default, the exit status of jobber relates only to the operation environment, with the indiviual command exit values being recorded in the database. Hence, if nothing breaks in the machinery of jobber itself, the exit status is zero, successful, just as if the job file was a shell script with a final `true` and no `set -e` before. A value of 1 is returned for at least one failed command with the --fail or --soft-fail switches. Other values result from errors in usage or overall operation according to the definitions in sysexits.h.
A simple scenario of running three tasks of hello world:
shell$ for n in 1 2 3; do echo "echo hello world $n"; done > job.sh
shell$ jobber job.sh init
[2022-02-25T17:48:23] appended 3 tasks
shell$ jobber job.sh
[2022-02-25T17:48:52] worker 1 task 1: echo hello world 1
hello world 1
[2022-02-25T17:48:52] worker 1 task 1 exit status: 0
shell$ jobber job.sh total done
3
1
shell$ jobber job.sh more && echo more to do || echo nothing more
more to do
shell$ jobber job.sh all
[2022-02-25T17:49:48] worker 2 task 2: echo hello world 2
hello world 2
[2022-02-25T17:49:48] worker 2 task 2 exit status: 0
[2022-02-25T17:49:48] worker 2 task 3: echo hello world 3
hello world 3
[2022-02-25T17:49:48] worker 2 task 3 exit status: 0
shell$ jobber job.sh total done queued
3
3
0
shell$ jobber job.sh more && echo more to do || echo nothing more
nothing moreparallel(1), xargs(1), make(1), ssh(1)
Thomas Orgis <thomas@orgis.org>