17. Autodoc Job Classes

A Job is a directory on disk containing input, source code, and output of a script execution. Objects of the Job class are used to represent jobs, providing helper functions to access the stored data.

There are two kinds of job classes, the Job class for finished jobs, and the CurrentJob for jobs that are currently executing. The latter has additional functionality that makes sense while executing, for example for creating new files.

class accelerator.Job(jobid, method=None)

A string that is a jobid, but also has some extra properties: .method The job method (can be the “name” when from build or urd). .number The job number as an int. .workdir The workdir name (the part before -number in the jobid) .path The filesystem directory where the job is stored. .params setup.json from this job. .post post.json from this job. .datasets list of Datasets in this job. And some functions: .withfile a JobWithFile with this job. .filename to join .path with a filename. .load to load a pickle. .json_load to load a json file. .open to open a file (like standard open) .dataset to get a named Dataset. .output to get what the job printed. .link_result to put a link in result_directory that points to a file in this job. .link_to_here to expose a subjob result in its parent.

Decays to a (unicode) string when pickled.

property build_job
chain(length=-1, reverse=False, stop_job=None)

Like Dataset.chain but for jobs.

dataset(name='default')
property datasets
filename(filename, sliceno=None)
files(pattern='*')
property input_directory
property is_build
json_load(filename='result.json', sliceno=None, unicode_as_utf8bytes=False, default=<object object>)

Put a symlink to filename in result_directory Only use this in a build script.

load(filename='result.pickle', sliceno=None, encoding='bytes', default=<object object>)

blob.load from this job

property method
number
open(filename, mode='r', sliceno=None, encoding=None, errors=None)
output(what=None)
property params
property parent
property path
property post
withfile(filename, sliced=False, extra=None)
workdir

Note

This list is not complete, see source code for details.

class accelerator.job.CurrentJob(jobid, params)

The currently running job (as passed to the method), with extra functions for writing data.

datasetwriter(columns={}, filename=None, hashlabel=None, hashlabel_override=False, caption=None, previous=None, name='default', parent=None, meta_only=False, for_single_slice=None, copy_mode=False, allow_missing_slices=False)
finish_early(result=None)

Finish job (successfully) without running later stages

input_directory
input_filename(*parts)
json_save(obj, filename='result.json', sliceno=None, sort_keys=True, temp=None, background=False)
open(filename, mode='r', sliceno=None, encoding=None, errors=None, temp=None)

Mostly like standard open with sliceno and temp, but you must use it as context manager with job.open(…) as fh: and the file will have a temp name until the with block ends.

open_input(filename, mode='r', encoding=None, errors=None)
register_file(filename)

Record a file produced by this job. Normally you would use job.open to have this happen automatically, but if the file was produced in a way where that is not practical you can use this to register it.

register_files(pattern='**/*')

Bulk register files matching a pattern. Tries to exclude internal files automatically. Does not register temp-files. The default pattern registers everything (recursively, unless python 2). Returns which files were registered.

save(obj, filename='result.pickle', sliceno=None, temp=None, background=False)