NAME

jobber - execute and manage a list of tasks as shell commands

SYNOPSIS

        jobber [options] <jobfile> [actions]

DESCRIPTION

This works on a job consisting of a number of tasks in a managed queue with filesystem-based synchronization for parallel instances possibly on multiple compute nodes. A task is simply a one-line shell command in the job text file (each line must end with a line break, including the last one). Lines commented via leading # are ignored. Continuation lines are not supported. The tasklist file name is mandatory, followed by one or more actions, which are executed in order if nothing fails:

init

initialize, create fresh queue and control files

all

run all tasks in queue (same as -1)

<number>

run this many tasks in sequence, -1 means all

exec[:<spec>]

execute specified tasks if still queued

stop

stop operation (do not start new tasks)

start

start operation (after stop)

more

exit with zero (true) if there is more to do right now

count[:<spec>]

print number of matching tasks (default: all)

queued

print number of tasks left in queue (count:queued)

running

print number of tasks being worked on (count:running)

done

print number of finished tasks (count:done)

failed

print number of failed tasks (count:failed)

skipped

print number of skipped tasks (count:skipped)

held

print number of held tasks (count:held)

times[:<spec>]

print grouped runtime stats

report[:<spec>]

print a tabular report

requeue[:<spec>]

put tasks back into queue (defaut: all)

redo

requeue done tasks (requeue:done)

retry

requeue failed tasks (requeue:failed,skipped)

rerun

requeue tasks marked as running (requeue:running)

(Make sure you do not requeue tasks with really active workers!)

hold[:<spec>]

hold specified queued tasks (default: all)

release[:<spec>]

release specified held tasks (default: all)

append

append task lines from standard input to the job

refresh

look for added tasks in job file

recover

try to restore queue file after worker death

cleanup

remove queue and control files

print tasks as shell script (default: all)

crc

compute CRC values for lines on stdin (job file ignored)

next

figure out and print next task

If the job file does not exist, the init action will create an empty one that you can fill using the append action.

A <spec> is a comma-separated list of the word 'all' to include all tasks, one of 'done', 'failed', 'held', 'queued', 'running', and 'skipped' to select tasks of that state, a numeric task ID or a range separated by a hyphen (a-b, or even a- for starting at task a and going to the end), or more explicitly <field>=<value> for fields state, id, ret, exit, signal, and crc. States may be appreviated with the first letter only. Multiple specifications select tasks that match any of them. The values for ret/exit/signal are only checked for tasks in failed or done states.

As long as a job is being worked on, the existing contents of the job file are expected not to change. Appending is OK, but best done explicitly via the synchronized jobber append operation. If you change existing commands, possibly moving offsets in the file while at that, chances are that you will confuse or at least irritate the workers and yourself later The workers will skip tasks with changed command lines (judged by CRC).

The refresh action is supposed to add tasks that have been appended to the job file after queue initialization. Existing commands are not checked for being changed. The workers do that before trying to execute and it would complicate state semantics if we interfered.

The exec action prepares a fixed list of tasks first and then starts executing them, ignoring further additions of egilible tasks at runtime. There lies the difference between all and exec:all (or just all and exec).

The print action without further arguments produces a shell script with all commands, but only those for queued tasks uncommented. Also, a header with some global stats is included int that case. A limited selection of tasks yields a shorter printout with all of them uncommented. Return values are parsed and actual exit status or kill signal presented for finished tasks.

The report table contains task and worker ID, runtime, start time, CRC and exit status or signal. A value of -1 is put in where there is no sensible value. Tasks running right now show a start time, but no runtime yet.

You can temporarily exempt tasks from execution (putting on hold)via the hold command. Releasing them restores the queued (pending) state.

The main process creates a child process for each task, possibly in parallel. Signals INT, HUP, and TERM in the main process trigger TERM to all children to end things gracefully, with the abortion of tasks redcorded in the queue file. Stdout and stderr output is serialized by the parent to produce intact lines. Text output is assumed. Binary data should be redirected in the task command lines. Standard input is passed through only for non-parallel work. =head1 PARAMETERS

These are the general rules for specifying parameters to this program:

        jobber -s -xyz -s=value --long --long=value [--] [files/stuff]

You mention the options to change parameters in any order or even multiple times. They are processed in the oder given, later operations overriding/extending earlier settings. Using the separator "--" stops option parsing An only mentioned short/long name (no "=value") means setting to 1, which is true in the logical sense. Also, prepending + instead of the usual - negates this, setting the value to 0 (false). Specifying "-s" and "--long" is the same as "-s=1" and "--long=1", while "+s" and "++long" is the sames as "-s=0" and "--long=0".

There are also different operators than just "=" available, notably ".=", "+=", "-=", "*=" and "/=" for concatenation / appending array/hash elements and scalar arithmetic operations on the value. Arrays are appended to via "array.=element", hash elements are set via "hash.=name=value". You can also set more array/hash elements by specifying a separator after the long parameter line like this for comma separation:

        --array/,/=1,2,3  --hash/,/=name=val,name2=val2

The available parameters are these, default values (in Perl-compatible syntax) at the time of generating this document following the long/short names:

config, I (array)
        []

Which configfile(s) to use (overriding automatic search in likely paths); special: just -I or --config causes printing a current config file to STDOUT

endtime, e (scalar)
        ''

time limit as a given point in time, overriding --time, value being a date specification that is just passed to date -d (use @123 to specify epoch 123, preferrably ISO date format otherwise)

exec-limit (scalar)
        2

End operation after too many execution attemps (i.e. bad shell, fork issues), number multiplied with current total task count, zero disables the limit for possibly endless looping on automatically requeued tasks.

fail, F (scalar)
        0

fail and stop execution on the first unsuccessful task

force (scalar)
        0

Do things as requested even if there are signs against that.

help, h (scalar)
        0

Show the help message. Value 1..9: help level, par: help for paramter par (long name) only.

Additional fun with negative values, optionally followed by comma-separated list of parameter names: -1: list par names, -2: list one line per name, -3: -2 without builtins, -10: dump values (Perl style), -11: dump values (lines), -100: print POD.

history, H (scalar)
        0

show execution history for task print and report (the latter dropping tasks not executed yet)

ioblock (scalar)
        16384

Block size for reading messages from worker processes.

iodir, d (scalar)
        ''

Put stdout/stderr of tasks into the specified directory (if non-empty), with file names <task>.<worker>.out and <task>.<worker>.err, read stdin from <task>.in (or /dev/null).

label, l (scalar)
        0

Label task stdout/stderr line printouts with the task ID (and the worker ID if value > 1).

lock-timeout (scalar)
        0

abort lock attempt after that many seconds, ending total operation

lock-warn (scalar)
        60

warn about locking taking over given amount of seconds

msgprefix (scalar)
        ''

prefix to print before jobber messages to distinguish them from your command output

noio, n (scalar)
        0

no standard intput/output/error at all to/from tasks (/dev/null)

parallel, p (scalar)
        1

Start that many processes for working on tasks in parallel, just like repeating the jobber command for multiple background tasks. The given count of actions is for each process.

pause (scalar)
        0

Pause for that many integer milliseconds before forking a worker (with locked queue).

quiet, q (scalar)
        0

no non-error messages about task execution (overrides verbosity)

shell, s (scalar)
        ''

Expicitly set the shell to use for the task commands. This is fixed in the queue file on init. It needs to accept -c "command line" but nothing else is required regarding its syntax. If none is specified here, a #! line in the job script will set the value, falling back to the SHELL environment variable and as last resort /bin/sh. That value is split once after the first whitespace to separate interpreter path and argument, just like it is customary for script parsing.

soft-fail, f (scalar)
        0

reflect unsuccessful tasks as overall non-zero exit value

sync (scalar)
        1

Use file flush and sync to safeguard changes to the job queue. Disable at your own peril.

time, t (scalar)
        'none'

time limit in seconds; stop running tasks if given time is running out ("none" disables time limit)

time-method (scalar)
        '2max'

how to estimate the expected runtime of a task, counting towards the limit, either a fixed number of seconds or a factor followed by one of min, mean, and max (like the default 2max) for using the minimal, mean, or maximal past task runtime as base

timestamp, T (scalar)
        1

prefix messages with a timestamp

undone, u (scalar)
        0

When printing all tasks, do not comment out command lines of tasks that have been executed.

verbose, v (scalar)
        0

increase/set verbosity level (up to 3)

version (scalar)
        0

print out the program version

RATIONALE

The motivation for this tool was the need to work on a predetermined set of independent computations with a scientific simulation program covering parts of some parameter space. Things should be documented and reproducable. Parts that fail should be fixable and then re-startable without re-running everything. There might be differing instances of worker processes on multiple compute servers picking tasks at the same time. One approach to such a problem are array jobs in batch systems, where you need to decide on a mapping of array ID to parameter set when setting up the job. Pre-generating the program configurations as shell command lines decouples the decision what has to be computed from how it will be distributed on some cluster and put into action. It also helps manual testing and reproduction outside the batch environment.

Using jobber, you have a simple sub-scheduler that can make better use of your batch system time slots, pushing through a number of your individual computations of uncertain (sometimes very short, sometimes longer) duration where otherwise an array job might have to be rescheduled for each small one. The clear recording of the success of each job task avoids you having to sift through log files and matching up batch system job IDs to the actual tasks computed. There is some othogonality to exploit, having the batch system distribute your work, deciding when and where it runs, and having jobber decide and record what gets actually computed. When you have some thousands of tasks to get through, with expected failures and restarts along the way, this can be rather helpful.

Outside the realm of batch systems, there are other tools that mainly help you running a number of tasks in parallel in an ad-hoc fashion. If you have a common command and a set of arguments to distribute to instances of that command, and just want to get through them without much fuss you might be happy using xargs or GNU/moreutils parallel. A strength of those tools is that they can construct the command lines to run on the fly. This is in contrast to a situation where you might spend days figuring out the correct commands and arguments to use, iterating over details with test runs and refining the final set of tasks.

A Make-like tool is a better solution when your tasks have interdependencies and the correct execution order along with possible parallelism need to be figured out from your description of sources and targets. This does not have to relate to building software. Any kind of file can be a Make target. A Makefile also provides documentation and reproducibility, like a flat shell script serving as input to jobber.

Jobber can be used to directly run things in parallel in one instance, but also many jobber instances can be started independently on differing computers to work together on the job queue on a shared POSIX filesystem. The latter needs to provide atomic rename and fsync, which is a bit more common than proper distributed locking (which you might have neutered with NFS mount options). The point is that you run the same command everywhere, e.g.

        jobber /path/to/jobfile all

to run the next tasks from the queue, arbitrarily prepared beforehand. Jobber does not distribute the work among networked machines itself. If you want work to happen on a remote box, you either arrange for the jobber instances being started where desired or just put ssh into the command lines to run one instance as a central dispatcher. You got the power of the shell to play any tricks there, jobber does not care.

QUEUE FILE FORMAT

Jobber creates and works on a queue file that contains pointers to the command lines in the job file. You are responsible for not changing the latter in ways that confuse operation. As a safeguard to spot modifications that would result in corrupted command lines, checksums of the commands are verified before executing them. If the checksum does not match, the task is marked as skipped instead.

The file format is designed to be somewhat readable as plain text but also to enable recording of task execution status in-place without having to move other file contents around. This means that header fields are padded to expect possible values (20 decimal digits for 64 bits), as is the return/exit code field for a task. Choices have been made to avoid unnecessarily long lines for each task entry. An adequate CRC16 value for modification checks instead of some long cryptographically secure hash is one such choice.

Example

An example with a few tasks:

        JBQv2   /bin/bash
        cnt     00000000000000000003
        twk     00000000000000000001
        tdn     00000000000000000001
        tfl     00000000000000000001
        tsk     00000000000000000000
        nwk     00000000000000000002
        mxt     00000000000000000452
        mnt     00000000000000000321
        smc     00000000000000000002
        smt     00000000000000000773
        nxt     00000000000000000266
        Q       1       130     12804   00000
        W       2       161     34222   00000
        D       3       234     00783   00000
        F       4       313     12364   00001
        B       2       1633875385      2
        B       3       1633875386      3
        E       3       1633875796      3       0
        B       4       1633875796      4
        E       4       1633876158      4       1

All fields are separated with tabs to ease ad-hoc parsing using grep and cut. As plain text, the file format is portable between different types of machines, as long as their execution environment agrees on the line ending and alphanumerical encoding.

Explanation

It begins with a file type identifier and the shell the command lines are supposed to be executed with, usually taken from the environment at queue creation time or, if the first line in the job file looked like it, from there. Directly after that the queue statistics lines follow, with these numerical fields, each on one line:

cnt

Total count of job tasks.

twk

Job tasks being worked on (marked as running).

tdn

Successfully finished job tasks.

tfl

Failed job tasks.

tsk

Skipped job tasks, be it because of some inconsistency (CRC failure) or a possibly temporary issue forking/launching the shell.

nwk

The number of workers workers for this queue. Each one increments that value and then has a personal ID to write into lines. Workers ending operations do _not_ decrement the value, so that it stays unique. There is one worker started per task, so that the combination of task ID and worker ID (reported as task.worker in diagnostic messages) is unique for the whole job run up to the next init.

The worker ID is basically a counter for task runs, be it different tasks or the same task repeatedly.

mxt

Maximum recorded task execution time in seconds.

mnt

Minimum recorded task execution time in seconds.

smc

Job task count that went into the execution time sum. This includes all task runs that went into accumulating the overall execution time, possibly inflated by requeueing tasks. The purpose is to get a meaningful mean execution time when dividing the sum.

smt

Sum of all execution times in seconds.

nxt

Whole-file byte offset of next entry to work on. To avoid reading through the whole file all the time, the next entry's offset is stored. Each operation that modifies task state needs to update this. A value of zero indicates that there is no known next task and a worker should indeed parse through the whole file to find it. An obviously invalid value is treated the same way. The search continues from the indicated position on and then re-starts from the beginning if the value was non-zero.

The times are initialized with zero. If they are still zero when the count of done or failed tasks is non-zero, this indeed means that the tasks finished in under one second.

After this header the task state records and the execution log lines follow. Each command line in the job file translates to one task ID (job file line counter, disregarding comment lines). Each of the state and log lines starts with an uppercase letter followed by a set of numbers. The first number always is the task ID.

The task state records are led by the letters Q, W, D, F, or S with these meanings:

Q (queued)

The task is in the queue, ready for execution.

W (running)

It is in work currently.

D (done)

The work is done, the task finished with good outcome.

F (failed)

The task execution failed (details in the return value field).

S (skipped)

The task was skipped. The only proper reason for that right now is failure to verify the CRC of the command line (changed job file).

H (held)

The task ist on hold. It can be scheduled for execution by releasing it into the queue again.

A full state record is comprised of the fields

        <state> <id>    <cmd>   <crc>   <ret>

with these meanings:

state

The state code, see above. It is modified in-place when execution state changes.

cmd

Job file offset to locate command line.

crc

A CRC-16 value of the command line. Execution is not attempted if that does not match. This field is of fixed width 5.

ret

The 16-bit return value following the waitpid() format ($? in Perl), also fixed to width 5. It starts out as all zeroes and is updated in-place after execution.

The log entries begin with uppercase letters with these meanings:

B

Execution of task began.

E

Execution of task ended. The 16-bit return value of this attempt is also appended to the line.

R

The task is being rejected by the worker and marked as skipped.

A

The task execution was aborted without doing any work, the task having been requeued. This happens when the actual shell execution did not work.

Each log entry is completed by a timestamp (epoch seconds) and worker number, and the 16-bit return value from this attempt.

The lines

        B       3       1633875386      2
        E       3       1633875796      2       0

from the example above inform you about task 3 being started by worker 2 at 2021-10-10T14:16:26 UTC and execution ending at 2021-10-10T14:23:16.

These log entries are appended as operations go on without disturbing the task state records that are by construction preceeding the log entries. Note that state records and log lines can still be intermixed as new tasks are added to the queue.

EXIT STATUS

By default, the exit status of jobber relates only to the operation environment, with the indiviual command exit values being recorded in the database. Hence, if nothing breaks in the machinery of jobber itself, the exit status is zero, successful, just as if the job file was a shell script with a final `true` and no `set -e` before. A value of 1 is returned for at least one failed command with the --fail or --soft-fail switches. Other values result from errors in usage or overall operation according to the definitions in sysexits.h.

EXAMPLES

A simple scenario of running three tasks of hello world:

        shell$ for n in 1 2 3; do echo "echo hello world $n"; done > job.sh
        shell$ jobber job.sh init
        [2022-02-25T17:48:23] appended 3 tasks
        shell$ jobber job.sh
        [2022-02-25T17:48:52] worker 1 task 1: echo hello world 1
        hello world 1
        [2022-02-25T17:48:52] worker 1 task 1 exit status: 0
        shell$ jobber job.sh total done
        3
        1
        shell$ jobber job.sh more && echo more to do || echo nothing more
        more to do
        shell$ jobber job.sh all
        [2022-02-25T17:49:48] worker 2 task 2: echo hello world 2
        hello world 2
        [2022-02-25T17:49:48] worker 2 task 2 exit status: 0
        [2022-02-25T17:49:48] worker 2 task 3: echo hello world 3
        hello world 3
        [2022-02-25T17:49:48] worker 2 task 3 exit status: 0
        shell$ jobber job.sh total done queued
        3
        3
        0
        shell$ jobber job.sh more && echo more to do || echo nothing more
        nothing more

SEE ALSO

parallel(1), xargs(1), make(1), ssh(1)

AUTHOR

Thomas Orgis <thomas@orgis.org>