jobcontrol.core

Objects responsible for JobControl core functionality.

Note

Important objects from this module should be imported in main __init___, in order to “abstract away” the namespace and have them in a more nicely accessible place.

class jobcontrol.core.JobControl(storage, config)

The main JobControl class.

Parameters:

• storage – A valid storage for the builds state. Must be an instance of a jobcontrol.interfaces.StorageBase subclass (or a compatible one).
• config – A jobcontrol.config.JobControlConfig instance, or a dict which will be passed to that class constructor.

classmethod from_config_file(config_file)

Initialize JobControl by loading configuration from a file. Will also initialize storage taking values from the configuration.

Parameters:config_file – Path to configuration file, or an open file descriptor (or file-like object). Returns:a JobControl instance

classmethod from_config(config)

Initialize JobControl from some configuration.

Parameters:config – Either a jobcontrol.config.JobControlConfig instance, or a dict to be passed as argument to that class constructor. Returns:a JobControl instance

get_job(job_id)

Get a job, by id.

Parameters:job_id – The job id Returns:a JobInfo class instance associated with the requested job. Raises:jobcontrol.exceptions.NotFound if a job with that id was not found in the configuration.

iter_jobs()

Generator yielding all the jobs, one by one.

Yields:for each job, a JobInfo class instance associated with the job.

get_build(build_id)

Get a build, by id.

Parameters:build_id – The build id Returns:a BuildInfo instance associated with the build. Raises:jobcontrol.exceptions.NotFound if a build with that id was not found in the configuration.

create_build(job_id)

Create a build, from a job configuration.

Note

Currently, we require that all the dependencies have already been built; in the future, it will be possible to build them automatically.

Note

Also, current implementation doesn’t allow for customizations to either the job configuration nor the build one (pinning, dep/revdep building, ...).

Parameters:

job_id – Id of the job for which to start a build

Returns:

a BuildInfo instance associated with the newly created build.

Raises:

• jobcontrol.exceptions.NotFound if the specified job was not found.
• jobcontrol.exceptions.MissingDependencies if any required dependency has no successful build.

build_job(job_id)

Create and run a new build for the specified job.

This is simply a shortcut that runs create_build() then run_build(). (Mostly for compatibility reasons).

Returns:a BuildInfo instance associated with the newly created build.

run_build(build_id)

Actually run a build.

• take the build configuration
• make sure all the dependencies are built
• take return values from the dependencies -> pass as arguments
• run the build
• build the reverse dependencies as well, if required to do so

Parameters:build_id – either a BuildInfo instance, or a build id

prune_logs(policy=None)

report_progress(group_name, current, total, status_line='')

Report progress for the currently running build.

Parameters:

• group_name – The report “group name”: either a tuple representing the “path”, or None for the top-level.
• current – Current progress
• total – Total progress
• status_line – An optional line of text, describing the currently running operation.

get_celery_app()

Return the Celery application, configured with values from the current configuration.

Note

this is a bit hackish, as we are just updating configuration values in the global object with ones from the jobcontrol configuration, not replacing all the configuration at once.

class jobcontrol.core.JobExecutionContext(app, job_id, build_id)

Class to hold “global” context during job execution.

This class can also act as a context manager for temporary context:

with JobExecutionContext(app, job_id, build_id):
    pass # do stuff in an execution context

Parameters:

• app – The JobControl instance running jobs
• job_id – Id of the currently running job
• build_id – Id of the currently running build

push()

Push this context in the global stack

pop()

Pop this context from the global stack

current_app

Returns the currently running app

current_job

Returns a JobInfo instance associated with the currently running job.

current_build

Returns a BuildInfo instance associated with the currently running build.

class jobcontrol.core.JobControlLogHandler

Logging handler sending messages to the appropriate JobControl instance that will dispatch them to storage.

flush()

No-op, as we don’t need to flush anything

emit(record)

“Emit” the log record (if there is an execution context, store the log record appropriately; otherwise, just ignore it).

class jobcontrol.core.JobInfo(app, job_id, config)

High-level interface to jobs.

Provides high-level methods for, eg, creating a build out of this job or iterating builds.

Configuration is stored in the config attribute (a BuildConfig instance).

id

config

get_deps()

Iterate over jobs this job depends upon.

Yields:JobInfo instances

get_status()

Return a label describing the current status of the job.

Returns:

• 'not_built' the job has no builds
• 'running' the job has running builds
• 'success' the job has at least a successful build
• 'failed' the job only has failed builds
• 'outdated' the job has at least a successful build, but older than one dependency build

get_revdeps()

Iterate over jobs depending on this one

Yields:JobInfo instances

iter_builds(*a, **kw)

Iterate over builds for this job.

Accepts the same arguments as jobcontrol.interfaces.StorageBase.get_job_builds()

Yields:BuildInfo instances

get_builds(*a, **kw)

DEPRECATED alias for iter_builds()

run()

Trigger run for this job (will automatically create a build, etc.)

create_build()

get_latest_successful_build()

Get latest successful build for this job, if any. Otherwise, returns None.

get_docs()

Get documentation for this job.

get_conf_as_yaml()

Return the job configuration as serialized YAML, mostly for displaying on user interfaces.

has_builds()

Check whether this job has any build.

has_successful_builds()

Check whether this job has any successful build.

has_running_builds()

Check whether this job has any running build.

is_outdated()

Check whether any dependency has builds more recent than the newest build for this job.

can_be_built()

Checks whether a job can be built, i.e.: whether all the dependencies have at least one successful build.

class jobcontrol.core.BuildInfo(app, build_id, info=None)

High-level interface to builds.

Parameters:

• app – The JobControl instance this build was retrieved from
• build_id – The build id
• info – Optionally, this can be used to pre-populate the build information (useful, eg. if we are retrieving a bunch of builds from the database at once).

app

build_id

id

The build id

job_id

The job id

info

Property used to lazily access the build attributes.

Returns a dict with the following keys:

• 'id'
• 'job_id'
• 'start_time'
• 'end_time'
• 'started'
• 'finished'
• 'success'
• 'skipped'
• 'config'
• 'retval'
• 'exception'
• 'exception_tb'

config

descriptive_status

Return a label describing the current status of the build.

Returns:

• 'CREATED' if the build was not started yet
• 'RUNNING' if the build was started but did not finish
• 'SUCCESSFUL' if the build run with success
• 'SKIPPED' if the build was skipped
• 'FAILED' if the build execution failed

refresh()

Refresh the build status information from database

get_progress_info()

Get information about the build progress

get_job()

Get a JobInfo associated with this build’s job

delete()

Delete all information related to this build from database

run()

Calls run_build() on the main app for this build

iter_log_messages(**kw)

Iterate over log messages for this build.

Keywords are passed directly to the underlying iter_log_messages() method of the storage.