Concepts and terminology

Task-manager use several concepts which are explained here. The main thing to remember is that task-manager works differently than traditional python script. To simplify, we can consider a python script like a collections of procedures (like functions) and a succession of procedures calls and result processing. The task-manager works differently as you will just declare the procedures and the task-manager will manage itself the procedures calls, passing some outputs as inputs for other procedures, etc...

Tasks/Steps

In task-manager, you will define your code in entity called Tasks. You can see them as python functions. The main difference with traditional python scripts is that you will not call them directly, it’s the job of the task-manager. Tasks are defined by subclassing Task class (see Create your own tasks and recipe).

As you will not call directly the task, you must give some informations about which arguments do you need for calling the task. These informations is classifier of arguments. In example, an Item classified as directory must contains the path corresponding to the directory and an Item classified as project must contains the project name, the project version, ...

For each argument of your Task, you must define which classifier it accepts. In example, if you have this Task:

class SimpleTask(Task):

    def __call__(self, arg1, arg2):
        pass

You must define a mapping like this:

{"arg1" : "classifier1", "arg2" : "classifier2"}

For more explanations about how to create your task, see Create your own tasks and recipe.

Tasks are logical code, but steps are part of the execution. This difference is necessary for making recipes the more flexible. In example, you can have one task for extracting zip archive and one for extracting tar archive. You have two solutions, depending on you use step or tasks:

• If you use task directly, you must make a recipe for zip archive and another for tar archive and choose which one you use before launching the execution.
• If you use step, you make a recipe with a Step’s category “extract” and it means that you will use one task of category “extract” (Zip archive extraction or tar archive extraction).

Using step allow you to create more flexible recipe and reduce you work.

Steps also includes a mapping of arguments used to initialize tasks, it means that each task of the same category must accepts same arguments (see Create your own tasks and recipe for more informations about it).

Inputs/Outputs/Item

As you don’t manage the task execution, you can’t manage communication between them. This is where Items comes, they’re dedicated to communication between tasks. Tasks can consume and produce Items. Consumption when Items are given to task at execution, these Items are called Inputs of this task. Production when Items are returned by task execution, these Items are called Output of this task.

There is also some special Items called Base-inputs given at the execution, required to execute the first step.

Each Item contains several informations:

• Classifiers: An Item may have one or multiple classifiers. (classifiers can be “archive”, “sdist” etc.)
• Data: Each item is enclosing some data. An archive item (item which is classified as an “archive”) should have a path and a format identifier.
• Tags: A list of tags (this will be described below).

Classifiers

Classifiers are the corner stone of the execution order. You must see classifiers as something which answer this question “What represents this Item?”.

Classifiers are hierarchical, for example an Item classified as “file.archive” is also classified “file”. Items can have multiples items not linked hierarchically, for example “file” and “project.install_file”.

Data

Each item should contains some data that characterizes itself. For instance, an item classified as an “archive” should have a path and a format identifier.

This data is stored as a mapping (key/value data type).

Tags

Each item can be tagged with arbitrary identifiers, you can choose what you want , they have only the meaning you give them.

For more informations about tags and their utilization, take a look at More control over the execution flow.

Status

Each recipe’s step has a status indicating if the step has been executed and/or the result of the execution. These status are:

• MISSING: No appropriate Task found
• SKIPPED: Task was not executed
• KILLED: Task have been killed during it’s execution
• ERROR: An Error prevented the task to complete
• TOO_MUCH_ITMS: Multiple items can be selected for running the task
• FAILURE: Task executed but report failure
• NODATA: Task did not found the data it expected to find
• PARTIAL: Task executed partially
• SUCCESS: Task executed successfully

Recipe/execution

In order to be able to execute something, a recipe must be written. The recipe is simple, it list which step should be executed. As step is an abstraction of which tasks will be executed, recipe must also list which task are available.

Recipe file is written in JSON. The format is following:

{"recipe" :
        [{"task-category" : "install"}]
 "tasks" :
        ["install.InstallTask"]}

In the “recipe” field, we list steps, each mapping represents a step.

In the “tasks” filed, we list task, each element is the import path of a task.

Scheduler

Resume

Execution = recipe + base_inputs

Step = task(initializers)