duplicity.asyncscheduler module

Asynchronous job scheduler, for concurrent execution with minimalistic dependency guarantees.

class duplicity.asyncscheduler.AsyncScheduler(concurrency)

Bases: object

Easy-to-use scheduler of function calls to be executed concurrently. A very simple dependency mechanism exists in the form of barriers (see insert_barrier()).

Each instance has a concurrency level associated with it. A concurrency of 0 implies that all tasks will be executed synchronously when scheduled. A concurrency of 1 indicates that a task will be executed asynchronously, but never concurrently with other tasks. Both 0 and 1 guarantee strict ordering among all tasks (i.e., they will be executed in the order scheduled).

At concurrency levels above 1, the tasks will end up being executed in an order undetermined except insofar as is enforced by calls to insert_barrier().

An AsynchScheduler should be created for any independent process; the scheduler will assume that if any background job fails (raises an exception), it makes further work moot.

__execute_caller(caller)

__init__(concurrency)

Create an asynchronous scheduler that executes jobs with the given level of concurrency.

__run_asynchronously(fn, params)

__run_synchronously(fn, params)

__start_worker(caller)

Start a new worker.

insert_barrier()

Proclaim that any tasks scheduled prior to the call to this method MUST be executed prior to any tasks scheduled after the call to this method.

The intended use case is that if task B depends on A, a barrier must be inserted in between to guarantee that A happens before B.

schedule_task(fn, params)

Schedule the given task (callable, typically function) for execution. Pass the given parameters to the function when calling it. Returns a callable which can optionally be used to wait for the task to complete, either by returning its return value or by propagating any exception raised by said task.

This method may block or return immediately, depending on the configuration and state of the scheduler.

This method may also raise an exception in order to trigger failures early, if the task (if run synchronously) or a previous task has already failed.

NOTE: Pay particular attention to the scope in which this is called. In particular, since it will execute concurrently in the background, assuming fn is a closure, any variables used most be properly bound in the closure. This is the reason for the convenience feature of being able to give parameters to the call, to avoid having to wrap the call itself in a function in order to “fixate” variables in, for example, an enclosing loop.

wait()

Wait for the scheduler to become entirely empty (i.e., all tasks having run to completion).

IMPORTANT: This is only useful with a single caller scheduling tasks, such that no call to schedule_task() is currently in progress or may happen subsequently to the call to wait().